Note: This is test shard 2 of 8.
[==========] Running 8 tests from 6 test suites.
[----------] Global test environment set-up.
[----------] 1 test from RebalanceIgnoredTserversTest
[ RUN ] RebalanceIgnoredTserversTest.Basic
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20240503 02:11:22.752120 4117 test_util.cc:268] Using random seed: 1014333498
I20240503 02:11:22.766088 4117 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:22.766203 4117 ts_itest-base.cc:113] --------------
I20240503 02:11:22.766250 4117 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:11:22.766337 4117 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:22.766412 4117 ts_itest-base.cc:116] --------------
2024-05-03T02:11:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:22Z Disabled control of system clock
I20240503 02:11:22.797192 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.5.126:42901
--webserver_interface=127.4.5.126
--webserver_port=0
--builtin_ntp_servers=127.4.5.84:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.5.126:42901
--enable_range_replica_placement=false with env {}
W20240503 02:11:23.198858 4129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:23.199332 4129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:23.199493 4129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:23.207672 4129 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:23.207849 4129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:23.207934 4129 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:23.208025 4129 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:23.217820 4129 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.4.5.126:42901
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.5.126:42901
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.5.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:23.219620 4129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:23.221828 4129 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:23.235097 4135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.235570 4134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.236083 4137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.236778 4129 server_base.cc:1034] running on GCE node
I20240503 02:11:23.237884 4129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:23.239954 4129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:23.241194 4129 hybrid_clock.cc:648] HybridClock initialized: now 1714702283241118 us; error 53 us; skew 500 ppm
I20240503 02:11:23.241606 4129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:23.253662 4129 webserver.cc:460] Webserver started at http://127.4.5.126:39067/ using document root <none> and password file <none>
I20240503 02:11:23.254786 4129 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:23.254977 4129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:23.255415 4129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:23.259121 4129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "3293601ab440445f96b83304424d2904"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-zv3f"
I20240503 02:11:23.260154 4129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "3293601ab440445f96b83304424d2904"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-zv3f"
I20240503 02:11:23.267043 4129 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 02:11:23.271826 4143 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:23.273752 4129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240503 02:11:23.274008 4129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
uuid: "3293601ab440445f96b83304424d2904"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-zv3f"
I20240503 02:11:23.274274 4129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:23.297730 4129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:23.298859 4129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:23.299202 4129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:23.320900 4129 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.126:42901
I20240503 02:11:23.320915 4197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.126:42901 every 8 connection(s)
I20240503 02:11:23.322849 4129 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:23.328083 4198 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:23.331732 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 4129
I20240503 02:11:23.332130 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:23.343297 4198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904: Bootstrap starting.
I20240503 02:11:23.347203 4198 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:23.348520 4198 log.cc:824] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:23.351630 4198 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904: No bootstrap required, opened a new log
I20240503 02:11:23.361820 4198 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3293601ab440445f96b83304424d2904" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42901 } }
I20240503 02:11:23.362357 4198 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:23.362511 4198 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3293601ab440445f96b83304424d2904, State: Initialized, Role: FOLLOWER
I20240503 02:11:23.363476 4198 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3293601ab440445f96b83304424d2904" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42901 } }
I20240503 02:11:23.363791 4198 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:23.363950 4198 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:23.364159 4198 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:23.367460 4198 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3293601ab440445f96b83304424d2904" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42901 } }
I20240503 02:11:23.368117 4198 leader_election.cc:302] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3293601ab440445f96b83304424d2904; no voters:
I20240503 02:11:23.368877 4198 leader_election.cc:288] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:23.369096 4204 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:23.370903 4204 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [term 1 LEADER]: Becoming Leader. State: Replica: 3293601ab440445f96b83304424d2904, State: Running, Role: LEADER
I20240503 02:11:23.371554 4198 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:23.371573 4204 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3293601ab440445f96b83304424d2904" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42901 } }
I20240503 02:11:23.375123 4205 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3293601ab440445f96b83304424d2904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3293601ab440445f96b83304424d2904" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42901 } } }
I20240503 02:11:23.375566 4205 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.376224 4206 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3293601ab440445f96b83304424d2904. Latest consensus state: current_term: 1 leader_uuid: "3293601ab440445f96b83304424d2904" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3293601ab440445f96b83304424d2904" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42901 } } }
I20240503 02:11:23.376567 4206 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:23.377836 4213 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:23.384419 4213 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:23.398514 4213 catalog_manager.cc:1347] Generated new cluster ID: fc56442a43344fc9afb05e844a72bc27
I20240503 02:11:23.398708 4213 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:23.433825 4213 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:23.435393 4213 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:23.444612 4213 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 3293601ab440445f96b83304424d2904: Generated new TSK 0
I20240503 02:11:23.445745 4213 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:23.462168 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.65:0
--local_ip_for_outbound_sockets=127.4.5.65
--webserver_interface=127.4.5.65
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--builtin_ntp_servers=127.4.5.84:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
W20240503 02:11:23.936254 4223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:23.936697 4223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:23.936842 4223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:23.948588 4223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:23.948866 4223 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.65
I20240503 02:11:23.960613 4223 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.5.65
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.65
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:23.962486 4223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:23.964908 4223 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:23.980579 4228 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.980964 4229 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:23.981449 4231 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:23.982139 4223 server_base.cc:1034] running on GCE node
I20240503 02:11:23.982966 4223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:23.984251 4223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:23.985486 4223 hybrid_clock.cc:648] HybridClock initialized: now 1714702283985414 us; error 91 us; skew 500 ppm
I20240503 02:11:23.985853 4223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:23.990001 4223 webserver.cc:460] Webserver started at http://127.4.5.65:39245/ using document root <none> and password file <none>
I20240503 02:11:23.991147 4223 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:23.991443 4223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:23.991905 4223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:23.995937 4223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "7671450640a1493c8f22109d5af9b305"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-zv3f"
I20240503 02:11:23.997066 4223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "7671450640a1493c8f22109d5af9b305"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-zv3f"
I20240503 02:11:24.005518 4223 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.003s sys 0.004s
I20240503 02:11:24.011001 4237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.013015 4223 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:24.013290 4223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "7671450640a1493c8f22109d5af9b305"
format_stamp: "Formatted at 2024-05-03 02:11:23 on dist-test-slave-zv3f"
I20240503 02:11:24.013577 4223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:24.043848 4223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:24.045218 4223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:24.045694 4223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:24.047266 4223 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:24.049912 4223 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:24.050045 4223 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.050241 4223 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:24.050335 4223 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.124126 4223 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.65:43279
I20240503 02:11:24.124209 4352 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.65:43279 every 8 connection(s)
I20240503 02:11:24.126139 4223 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:24.130638 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 4223
I20240503 02:11:24.131019 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:24.135396 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.66:0
--local_ip_for_outbound_sockets=127.4.5.66
--webserver_interface=127.4.5.66
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--builtin_ntp_servers=127.4.5.84:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 02:11:24.147774 4353 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42901
I20240503 02:11:24.148303 4353 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:24.150071 4353 heartbeater.cc:511] Master 127.4.5.126:42901 requested a full tablet report, sending...
I20240503 02:11:24.153304 4161 ts_manager.cc:194] Registered new tserver with Master: 7671450640a1493c8f22109d5af9b305 (127.4.5.65:43279)
I20240503 02:11:24.155362 4161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.65:34147
W20240503 02:11:24.667240 4357 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:24.668516 4357 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:24.668659 4357 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:24.677043 4357 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:24.677268 4357 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.66
I20240503 02:11:24.688804 4357 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.5.66
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.66
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:24.690467 4357 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:24.692613 4357 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:24.708660 4362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.709105 4365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:24.709640 4363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:24.710251 4357 server_base.cc:1034] running on GCE node
I20240503 02:11:24.711035 4357 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:24.712146 4357 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:24.713384 4357 hybrid_clock.cc:648] HybridClock initialized: now 1714702284713294 us; error 65 us; skew 500 ppm
I20240503 02:11:24.713733 4357 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:24.716470 4357 webserver.cc:460] Webserver started at http://127.4.5.66:35471/ using document root <none> and password file <none>
I20240503 02:11:24.717372 4357 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:24.717517 4357 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:24.717866 4357 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:24.721140 4357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "a8728db0cfe34f6b8e821b541f25c8d3"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-zv3f"
I20240503 02:11:24.722092 4357 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "a8728db0cfe34f6b8e821b541f25c8d3"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-zv3f"
I20240503 02:11:24.728164 4357 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20240503 02:11:24.733186 4371 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.734824 4357 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:24.734997 4357 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "a8728db0cfe34f6b8e821b541f25c8d3"
format_stamp: "Formatted at 2024-05-03 02:11:24 on dist-test-slave-zv3f"
I20240503 02:11:24.735201 4357 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:24.753643 4357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:24.754746 4357 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:24.755079 4357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:24.756273 4357 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:24.759057 4357 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:24.759153 4357 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.759311 4357 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:24.759372 4357 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:24.804971 4357 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.66:40931
I20240503 02:11:24.805011 4485 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.66:40931 every 8 connection(s)
I20240503 02:11:24.806663 4357 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:24.807826 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 4357
I20240503 02:11:24.808027 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:24.812423 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.67:0
--local_ip_for_outbound_sockets=127.4.5.67
--webserver_interface=127.4.5.67
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--builtin_ntp_servers=127.4.5.84:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 02:11:24.824303 4486 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42901
I20240503 02:11:24.824697 4486 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:24.825673 4486 heartbeater.cc:511] Master 127.4.5.126:42901 requested a full tablet report, sending...
I20240503 02:11:24.827596 4161 ts_manager.cc:194] Registered new tserver with Master: a8728db0cfe34f6b8e821b541f25c8d3 (127.4.5.66:40931)
I20240503 02:11:24.829092 4161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.66:52347
I20240503 02:11:25.160621 4353 heartbeater.cc:503] Master 127.4.5.126:42901 was elected leader, sending a full tablet report...
W20240503 02:11:25.237119 4490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:25.237527 4490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:25.237648 4490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:25.246261 4490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:25.246507 4490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.67
I20240503 02:11:25.257601 4490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.5.67
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.67
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:25.259290 4490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:25.261333 4490 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:25.276859 4495 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.277381 4496 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.277848 4498 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:25.278498 4490 server_base.cc:1034] running on GCE node
I20240503 02:11:25.279232 4490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:25.280382 4490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:25.281643 4490 hybrid_clock.cc:648] HybridClock initialized: now 1714702285281558 us; error 63 us; skew 500 ppm
I20240503 02:11:25.282022 4490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:25.284759 4490 webserver.cc:460] Webserver started at http://127.4.5.67:45435/ using document root <none> and password file <none>
I20240503 02:11:25.285663 4490 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:25.285815 4490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:25.286156 4490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:25.289448 4490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "37f836e97ac541c6ac8b40b8f6d8dd03"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-zv3f"
I20240503 02:11:25.290280 4490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "37f836e97ac541c6ac8b40b8f6d8dd03"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-zv3f"
I20240503 02:11:25.297753 4490 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20240503 02:11:25.301795 4504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.303443 4490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20240503 02:11:25.303645 4490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "37f836e97ac541c6ac8b40b8f6d8dd03"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-zv3f"
I20240503 02:11:25.303862 4490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.322518 4490 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:25.324551 4490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:25.324903 4490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.326123 4490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:25.328254 4490 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:25.328346 4490 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.328509 4490 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:25.328570 4490 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20240503 02:11:25.371773 4490 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.67:35377
I20240503 02:11:25.371791 4617 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.67:35377 every 8 connection(s)
I20240503 02:11:25.373484 4490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:25.380153 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 4490
I20240503 02:11:25.380415 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:25.384141 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.68:0
--local_ip_for_outbound_sockets=127.4.5.68
--webserver_interface=127.4.5.68
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--builtin_ntp_servers=127.4.5.84:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 02:11:25.391562 4618 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42901
I20240503 02:11:25.392045 4618 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:25.393667 4618 heartbeater.cc:511] Master 127.4.5.126:42901 requested a full tablet report, sending...
I20240503 02:11:25.395757 4161 ts_manager.cc:194] Registered new tserver with Master: 37f836e97ac541c6ac8b40b8f6d8dd03 (127.4.5.67:35377)
I20240503 02:11:25.397097 4161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.67:56929
W20240503 02:11:25.809394 4622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:25.809782 4622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:25.809962 4622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:25.820490 4622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:25.820786 4622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.68
I20240503 02:11:25.832489 4486 heartbeater.cc:503] Master 127.4.5.126:42901 was elected leader, sending a full tablet report...
I20240503 02:11:25.836037 4622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.4.5.68
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.68
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:25.838810 4622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:25.841703 4622 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:25.862162 4627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.862458 4628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:25.863127 4630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:25.863749 4622 server_base.cc:1034] running on GCE node
I20240503 02:11:25.864626 4622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:25.866214 4622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:25.867496 4622 hybrid_clock.cc:648] HybridClock initialized: now 1714702285867392 us; error 84 us; skew 500 ppm
I20240503 02:11:25.867980 4622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:25.875326 4622 webserver.cc:460] Webserver started at http://127.4.5.68:45327/ using document root <none> and password file <none>
I20240503 02:11:25.876773 4622 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:25.877032 4622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:25.877518 4622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:25.882694 4622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "acfb76aa5a9d4d1897242dd31b9e86e0"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-zv3f"
I20240503 02:11:25.884168 4622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "acfb76aa5a9d4d1897242dd31b9e86e0"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-zv3f"
I20240503 02:11:25.892851 4622 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20240503 02:11:25.896900 4637 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.898622 4622 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:11:25.898872 4622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "acfb76aa5a9d4d1897242dd31b9e86e0"
format_stamp: "Formatted at 2024-05-03 02:11:25 on dist-test-slave-zv3f"
I20240503 02:11:25.899101 4622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:25.936434 4622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:25.937633 4622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:25.937973 4622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:25.939134 4622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:25.941146 4622 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:25.941238 4622 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.941382 4622 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:25.942282 4622 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:25.985854 4622 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.68:45395
I20240503 02:11:25.985888 4750 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.68:45395 every 8 connection(s)
I20240503 02:11:25.987529 4622 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:25.992321 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 4622
I20240503 02:11:25.992564 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:25.996744 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.69:0
--local_ip_for_outbound_sockets=127.4.5.69
--webserver_interface=127.4.5.69
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--builtin_ntp_servers=127.4.5.84:38143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--consensus_rpc_timeout_ms=30000 with env {}
I20240503 02:11:26.005985 4751 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42901
I20240503 02:11:26.006390 4751 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:26.007468 4751 heartbeater.cc:511] Master 127.4.5.126:42901 requested a full tablet report, sending...
I20240503 02:11:26.009241 4161 ts_manager.cc:194] Registered new tserver with Master: acfb76aa5a9d4d1897242dd31b9e86e0 (127.4.5.68:45395)
I20240503 02:11:26.010365 4161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.68:56521
I20240503 02:11:26.401175 4618 heartbeater.cc:503] Master 127.4.5.126:42901 was elected leader, sending a full tablet report...
W20240503 02:11:26.431842 4755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:26.432343 4755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:26.432526 4755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:26.441437 4755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:26.441735 4755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.69
I20240503 02:11:26.453451 4755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_cache_size_limit_mb=10
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.4.5.69
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42901
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.69
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:26.455443 4755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:26.457828 4755 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:26.473614 4760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.474062 4761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:26.474570 4763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:26.475215 4755 server_base.cc:1034] running on GCE node
I20240503 02:11:26.475986 4755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:26.477257 4755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:26.478518 4755 hybrid_clock.cc:648] HybridClock initialized: now 1714702286478420 us; error 52 us; skew 500 ppm
I20240503 02:11:26.478916 4755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:26.482069 4755 webserver.cc:460] Webserver started at http://127.4.5.69:39371/ using document root <none> and password file <none>
I20240503 02:11:26.483078 4755 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:26.483292 4755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:26.483682 4755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:26.487150 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e4877c98f2b640d390b4bce294a8ac1b"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-zv3f"
I20240503 02:11:26.488058 4755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e4877c98f2b640d390b4bce294a8ac1b"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-zv3f"
I20240503 02:11:26.494743 4755 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20240503 02:11:26.499447 4769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.501382 4755 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20240503 02:11:26.501590 4755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e4877c98f2b640d390b4bce294a8ac1b"
format_stamp: "Formatted at 2024-05-03 02:11:26 on dist-test-slave-zv3f"
I20240503 02:11:26.501822 4755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:26.520740 4755 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:26.521865 4755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:26.522248 4755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:26.523406 4755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:26.525259 4755 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:26.525339 4755 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.525481 4755 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:26.525534 4755 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:26.570065 4755 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.69:43055
I20240503 02:11:26.570093 4883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.69:43055 every 8 connection(s)
I20240503 02:11:26.571828 4755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:26.582041 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 4755
I20240503 02:11:26.582304 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceIgnoredTserversTest.Basic.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:26.588398 4884 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42901
I20240503 02:11:26.588826 4884 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:26.590711 4884 heartbeater.cc:511] Master 127.4.5.126:42901 requested a full tablet report, sending...
I20240503 02:11:26.592516 4161 ts_manager.cc:194] Registered new tserver with Master: e4877c98f2b640d390b4bce294a8ac1b (127.4.5.69:43055)
I20240503 02:11:26.593508 4161 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.69:53047
I20240503 02:11:26.600633 4117 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:26.626982 4161 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:11:26.672698 4553 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca53b3d16744385945c87651913ec27 (DEFAULT_TABLE table=TestTable [id=43f48b053d7a4d97b1ab1d1d62204385]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:26.674077 4686 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca53b3d16744385945c87651913ec27 (DEFAULT_TABLE table=TestTable [id=43f48b053d7a4d97b1ab1d1d62204385]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:26.674784 4553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca53b3d16744385945c87651913ec27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:26.675875 4686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca53b3d16744385945c87651913ec27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:26.678376 4420 tablet_service.cc:1467] Processing CreateTablet for tablet 1ca53b3d16744385945c87651913ec27 (DEFAULT_TABLE table=TestTable [id=43f48b053d7a4d97b1ab1d1d62204385]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:26.679803 4420 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ca53b3d16744385945c87651913ec27. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:26.693562 4904 tablet_bootstrap.cc:492] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0: Bootstrap starting.
I20240503 02:11:26.696750 4905 tablet_bootstrap.cc:492] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3: Bootstrap starting.
I20240503 02:11:26.698364 4903 tablet_bootstrap.cc:492] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03: Bootstrap starting.
I20240503 02:11:26.700500 4904 tablet_bootstrap.cc:654] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:26.700680 4905 tablet_bootstrap.cc:654] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:26.702193 4903 tablet_bootstrap.cc:654] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:26.702139 4904 log.cc:824] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:26.702627 4905 log.cc:824] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:26.703486 4903 log.cc:824] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:26.705883 4904 tablet_bootstrap.cc:492] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0: No bootstrap required, opened a new log
I20240503 02:11:26.706245 4904 ts_tablet_manager.cc:1367] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240503 02:11:26.706429 4905 tablet_bootstrap.cc:492] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3: No bootstrap required, opened a new log
I20240503 02:11:26.706702 4905 ts_tablet_manager.cc:1367] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20240503 02:11:26.708315 4903 tablet_bootstrap.cc:492] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03: No bootstrap required, opened a new log
I20240503 02:11:26.708698 4903 ts_tablet_manager.cc:1367] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20240503 02:11:26.714697 4905 raft_consensus.cc:348] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.715180 4905 raft_consensus.cc:374] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:26.715317 4905 raft_consensus.cc:729] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8728db0cfe34f6b8e821b541f25c8d3, State: Initialized, Role: FOLLOWER
I20240503 02:11:26.716346 4905 consensus_queue.cc:260] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.716336 4904 raft_consensus.cc:348] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.716962 4904 raft_consensus.cc:374] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:26.717132 4904 raft_consensus.cc:729] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: acfb76aa5a9d4d1897242dd31b9e86e0, State: Initialized, Role: FOLLOWER
I20240503 02:11:26.718061 4904 consensus_queue.cc:260] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.718479 4903 raft_consensus.cc:348] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.719100 4903 raft_consensus.cc:374] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:26.719261 4903 raft_consensus.cc:729] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 37f836e97ac541c6ac8b40b8f6d8dd03, State: Initialized, Role: FOLLOWER
I20240503 02:11:26.720118 4903 consensus_queue.cc:260] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.720769 4751 heartbeater.cc:503] Master 127.4.5.126:42901 was elected leader, sending a full tablet report...
I20240503 02:11:26.720930 4905 ts_tablet_manager.cc:1398] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20240503 02:11:26.722873 4904 ts_tablet_manager.cc:1398] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
I20240503 02:11:26.723531 4903 ts_tablet_manager.cc:1398] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03: Time spent starting tablet: real 0.015s user 0.004s sys 0.009s
W20240503 02:11:26.740238 4752 tablet.cc:2369] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:26.809887 4487 tablet.cc:2369] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:26.876998 4619 tablet.cc:2369] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:26.933184 4909 raft_consensus.cc:482] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:26.933544 4909 raft_consensus.cc:504] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.935657 4909 leader_election.cc:288] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 37f836e97ac541c6ac8b40b8f6d8dd03 (127.4.5.67:35377), acfb76aa5a9d4d1897242dd31b9e86e0 (127.4.5.68:45395)
I20240503 02:11:26.945343 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca53b3d16744385945c87651913ec27" candidate_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" is_pre_election: true
I20240503 02:11:26.946063 4706 raft_consensus.cc:2454] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8728db0cfe34f6b8e821b541f25c8d3 in term 0.
I20240503 02:11:26.946092 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca53b3d16744385945c87651913ec27" candidate_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" is_pre_election: true
I20240503 02:11:26.946723 4573 raft_consensus.cc:2454] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a8728db0cfe34f6b8e821b541f25c8d3 in term 0.
I20240503 02:11:26.946791 4374 leader_election.cc:302] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a8728db0cfe34f6b8e821b541f25c8d3, acfb76aa5a9d4d1897242dd31b9e86e0; no voters:
I20240503 02:11:26.947247 4909 raft_consensus.cc:2781] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:26.947386 4909 raft_consensus.cc:482] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:26.947494 4909 raft_consensus.cc:3037] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:26.949975 4909 raft_consensus.cc:504] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.950603 4909 leader_election.cc:288] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [CANDIDATE]: Term 1 election: Requested vote from peers 37f836e97ac541c6ac8b40b8f6d8dd03 (127.4.5.67:35377), acfb76aa5a9d4d1897242dd31b9e86e0 (127.4.5.68:45395)
I20240503 02:11:26.951103 4573 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca53b3d16744385945c87651913ec27" candidate_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03"
I20240503 02:11:26.951156 4706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ca53b3d16744385945c87651913ec27" candidate_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0"
I20240503 02:11:26.951323 4573 raft_consensus.cc:3037] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:26.951362 4706 raft_consensus.cc:3037] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:26.953756 4706 raft_consensus.cc:2454] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8728db0cfe34f6b8e821b541f25c8d3 in term 1.
I20240503 02:11:26.953775 4573 raft_consensus.cc:2454] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a8728db0cfe34f6b8e821b541f25c8d3 in term 1.
I20240503 02:11:26.954217 4372 leader_election.cc:302] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 37f836e97ac541c6ac8b40b8f6d8dd03, a8728db0cfe34f6b8e821b541f25c8d3; no voters:
I20240503 02:11:26.954519 4909 raft_consensus.cc:2781] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:26.955024 4909 raft_consensus.cc:686] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [term 1 LEADER]: Becoming Leader. State: Replica: a8728db0cfe34f6b8e821b541f25c8d3, State: Running, Role: LEADER
I20240503 02:11:26.955608 4909 consensus_queue.cc:237] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:26.962996 4160 catalog_manager.cc:5484] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 reported cstate change: term changed from 0 to 1, leader changed from <none> to a8728db0cfe34f6b8e821b541f25c8d3 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:27.045980 4117 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:27.049453 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 7671450640a1493c8f22109d5af9b305 to finish bootstrapping
I20240503 02:11:27.062636 4117 ts_itest-base.cc:233] Waiting for 1 tablets on tserver a8728db0cfe34f6b8e821b541f25c8d3 to finish bootstrapping
I20240503 02:11:27.072396 4117 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 37f836e97ac541c6ac8b40b8f6d8dd03 to finish bootstrapping
I20240503 02:11:27.082819 4117 ts_itest-base.cc:233] Waiting for 1 tablets on tserver acfb76aa5a9d4d1897242dd31b9e86e0 to finish bootstrapping
I20240503 02:11:27.091857 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e4877c98f2b640d390b4bce294a8ac1b to finish bootstrapping
I20240503 02:11:27.373934 4909 consensus_queue.cc:1035] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:27.399912 4914 consensus_queue.cc:1035] T 1ca53b3d16744385945c87651913ec27 P a8728db0cfe34f6b8e821b541f25c8d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:11:27.596850 4884 heartbeater.cc:503] Master 127.4.5.126:42901 was elected leader, sending a full tablet report...
I20240503 02:11:27.736148 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.737632 4420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:27.749747 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.757901 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:27.793340 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.498060 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.501228 4420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:28.512722 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.531674 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.531915 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.697326 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.706801 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.708263 4420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:28.719405 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.727377 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.902298 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.913378 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.915005 4420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:28.918020 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:28.920962 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:29.541821 4160 ts_manager.cc:292] Set tserver state for 7671450640a1493c8f22109d5af9b305 to MAINTENANCE_MODE
I20240503 02:11:31.088933 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.094560 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.099891 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.108405 4420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:31.142282 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.291591 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.305918 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.306061 4420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:31.316242 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.333480 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.506925 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.512763 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.513484 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.536670 4420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:31.551330 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.695188 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.698444 4420 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:31.699520 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.701042 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.711561 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:31.886569 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 4357
I20240503 02:11:31.888170 4481 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:32.009184 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 4357
W20240503 02:11:32.029721 4639 connection.cc:535] server connection from 127.4.5.66:56823 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:32.580672 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.585803 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.592828 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.592912 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.736532 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.747453 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.747509 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.760636 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.929425 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.937912 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.940063 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:32.946923 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:33.096868 4553 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:33.099035 4287 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:33.102142 4819 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:33.103118 4686 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:33.281062 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 4223
I20240503 02:11:33.282874 4347 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.401543 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 4223
I20240503 02:11:33.423530 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 4490
I20240503 02:11:33.425043 4613 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.510330 5357 raft_consensus.cc:482] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a8728db0cfe34f6b8e821b541f25c8d3)
I20240503 02:11:33.510661 5357 raft_consensus.cc:504] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "37f836e97ac541c6ac8b40b8f6d8dd03" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 35377 } } peers { permanent_uuid: "acfb76aa5a9d4d1897242dd31b9e86e0" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 45395 } } peers { permanent_uuid: "a8728db0cfe34f6b8e821b541f25c8d3" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40931 } }
I20240503 02:11:33.512977 5357 leader_election.cc:288] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 37f836e97ac541c6ac8b40b8f6d8dd03 (127.4.5.67:35377), a8728db0cfe34f6b8e821b541f25c8d3 (127.4.5.66:40931)
W20240503 02:11:33.514945 4641 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.66:40931: connect: Connection refused (error 111)
W20240503 02:11:33.517046 4641 leader_election.cc:334] T 1ca53b3d16744385945c87651913ec27 P acfb76aa5a9d4d1897242dd31b9e86e0 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a8728db0cfe34f6b8e821b541f25c8d3 (127.4.5.66:40931): Network error: Client connection negotiation failed: client connection to 127.4.5.66:40931: connect: Connection refused (error 111)
I20240503 02:11:33.555473 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 4490
I20240503 02:11:33.555073 5361 raft_consensus.cc:482] T 1ca53b3d16744385945c87651913ec27 P 37f836e97ac541c6ac8b40b8f6d8dd03 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a8728db0cfe34f6b8e821b541f25c8d3)
I20240503 02:11:33.579900 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 4622
W20240503 02:11:33.581135 5359 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:11:33.513299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.5.67:35377 (local address 127.4.5.68:49691)
0503 02:11:33.514170 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0503 02:11:33.514200 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0503 02:11:33.514427 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:11:33.555655 (+ 41228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0503 02:11:33.555663 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0503 02:11:33.555711 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0503 02:11:33.556494 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0503 02:11:33.556520 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0503 02:11:33.579698 (+ 23178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.5.67:35377: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":635,"thread_start_us":777,"threads_started":1}
I20240503 02:11:33.581254 4745 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.708808 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 4622
I20240503 02:11:33.730362 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 4755
I20240503 02:11:33.731686 4878 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:33.851876 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 4755
I20240503 02:11:33.872298 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 4129
I20240503 02:11:33.873744 4193 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:34.027303 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 4129
2024-05-03T02:11:34Z chronyd exiting
[ OK ] RebalanceIgnoredTserversTest.Basic (11327 ms)
[----------] 1 test from RebalanceIgnoredTserversTest (11327 ms total)
[----------] 1 test from RebalanceStartCriteriaTest
[ RUN ] RebalanceStartCriteriaTest.TabletServerIsDown/1
I20240503 02:11:34.076515 4117 test_util.cc:268] Using random seed: 1025657843
I20240503 02:11:34.078503 4117 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:34.078610 4117 ts_itest-base.cc:113] --------------
I20240503 02:11:34.078672 4117 ts_itest-base.cc:114] 5 tablet servers
I20240503 02:11:34.078764 4117 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:11:34.078823 4117 ts_itest-base.cc:116] --------------
2024-05-03T02:11:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:34Z Disabled control of system clock
I20240503 02:11:34.107126 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.5.126:35993
--webserver_interface=127.4.5.126
--webserver_port=0
--builtin_ntp_servers=127.4.5.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.5.126:35993
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:11:34.490581 5372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:34.490996 5372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:34.491147 5372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:34.499367 5372 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:34.499533 5372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:34.499616 5372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:34.499722 5372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:34.499787 5372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:34.510193 5372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.4.5.126:35993
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.5.126:35993
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.5.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:34.511972 5372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:34.514108 5372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:34.527367 5378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:34.527961 5377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:34.528420 5380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:34.529157 5372 server_base.cc:1034] running on GCE node
I20240503 02:11:34.530203 5372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:34.532006 5372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:34.533252 5372 hybrid_clock.cc:648] HybridClock initialized: now 1714702294533209 us; error 54 us; skew 500 ppm
I20240503 02:11:34.533682 5372 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:34.542765 5372 webserver.cc:460] Webserver started at http://127.4.5.126:33649/ using document root <none> and password file <none>
I20240503 02:11:34.543810 5372 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:34.543954 5372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:34.544394 5372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:34.547783 5372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-zv3f"
I20240503 02:11:34.548792 5372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-zv3f"
I20240503 02:11:34.555121 5372 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 02:11:34.559262 5386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:34.560945 5372 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:11:34.561158 5372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9"
format_stamp: "Formatted at 2024-05-03 02:11:34 on dist-test-slave-zv3f"
I20240503 02:11:34.561398 5372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:34.582644 5372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:34.583742 5372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:34.584054 5372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:34.605491 5372 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.126:35993
I20240503 02:11:34.605512 5437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.126:35993 every 8 connection(s)
I20240503 02:11:34.607246 5372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:34.607651 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 5372
I20240503 02:11:34.607885 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:34.613025 5438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:34.625185 5438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9: Bootstrap starting.
I20240503 02:11:34.629312 5438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:34.630571 5438 log.cc:824] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:34.634008 5438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9: No bootstrap required, opened a new log
I20240503 02:11:34.642135 5438 raft_consensus.cc:348] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35993 } }
I20240503 02:11:34.642717 5438 raft_consensus.cc:374] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:34.642871 5438 raft_consensus.cc:729] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a28cbe80933e4f3f8d5fdc3964cc03b9, State: Initialized, Role: FOLLOWER
I20240503 02:11:34.643694 5438 consensus_queue.cc:260] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35993 } }
I20240503 02:11:34.643955 5438 raft_consensus.cc:388] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:34.644125 5438 raft_consensus.cc:482] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:34.644315 5438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:34.646891 5438 raft_consensus.cc:504] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35993 } }
I20240503 02:11:34.647532 5438 leader_election.cc:302] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a28cbe80933e4f3f8d5fdc3964cc03b9; no voters:
I20240503 02:11:34.648309 5438 leader_election.cc:288] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:34.648479 5443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:34.649339 5443 raft_consensus.cc:686] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [term 1 LEADER]: Becoming Leader. State: Replica: a28cbe80933e4f3f8d5fdc3964cc03b9, State: Running, Role: LEADER
I20240503 02:11:34.650066 5443 consensus_queue.cc:237] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35993 } }
I20240503 02:11:34.650998 5438 sys_catalog.cc:564] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:34.653604 5445 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a28cbe80933e4f3f8d5fdc3964cc03b9. Latest consensus state: current_term: 1 leader_uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35993 } } }
I20240503 02:11:34.653604 5444 sys_catalog.cc:455] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a28cbe80933e4f3f8d5fdc3964cc03b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35993 } } }
I20240503 02:11:34.654256 5445 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:34.654270 5444 sys_catalog.cc:458] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:34.657016 5448 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:34.663959 5448 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:34.674840 5448 catalog_manager.cc:1347] Generated new cluster ID: ac6a3c4071a44e1a92dcb9f8d490fb92
I20240503 02:11:34.675084 5448 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:34.702525 5448 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:34.704023 5448 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:34.717775 5448 catalog_manager.cc:5857] T 00000000000000000000000000000000 P a28cbe80933e4f3f8d5fdc3964cc03b9: Generated new TSK 0
I20240503 02:11:34.718957 5448 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:34.729089 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.65:0
--local_ip_for_outbound_sockets=127.4.5.65
--webserver_interface=127.4.5.65
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--builtin_ntp_servers=127.4.5.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:35.114763 5462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:35.115173 5462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:35.115322 5462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:35.124313 5462 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:35.124493 5462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:35.124696 5462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.65
I20240503 02:11:35.134954 5462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.5.65
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.65
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:35.136763 5462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:35.138952 5462 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:35.153916 5470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:35.154448 5468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:35.155004 5467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:35.155692 5462 server_base.cc:1034] running on GCE node
I20240503 02:11:35.156517 5462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:35.157784 5462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:35.159015 5462 hybrid_clock.cc:648] HybridClock initialized: now 1714702295158954 us; error 58 us; skew 500 ppm
I20240503 02:11:35.159418 5462 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:35.162210 5462 webserver.cc:460] Webserver started at http://127.4.5.65:46727/ using document root <none> and password file <none>
I20240503 02:11:35.163120 5462 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:35.163261 5462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:35.163635 5462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:35.166908 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "f3892a5f8dbc4c05bea4f419a4068d71"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-zv3f"
I20240503 02:11:35.167814 5462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "f3892a5f8dbc4c05bea4f419a4068d71"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-zv3f"
I20240503 02:11:35.173794 5462 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20240503 02:11:35.177773 5476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.179279 5462 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:11:35.179451 5462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "f3892a5f8dbc4c05bea4f419a4068d71"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-zv3f"
I20240503 02:11:35.179670 5462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:35.206460 5462 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:35.207528 5462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:35.207847 5462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:35.209041 5462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:35.210816 5462 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:35.210927 5462 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.211050 5462 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:35.211127 5462 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.246466 5462 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.65:35219
I20240503 02:11:35.246518 5590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.65:35219 every 8 connection(s)
I20240503 02:11:35.248104 5462 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:35.253545 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 5462
I20240503 02:11:35.253804 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:35.257959 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.66:0
--local_ip_for_outbound_sockets=127.4.5.66
--webserver_interface=127.4.5.66
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--builtin_ntp_servers=127.4.5.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:35.264830 5591 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35993
I20240503 02:11:35.265177 5591 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:35.266197 5591 heartbeater.cc:511] Master 127.4.5.126:35993 requested a full tablet report, sending...
I20240503 02:11:35.268748 5403 ts_manager.cc:194] Registered new tserver with Master: f3892a5f8dbc4c05bea4f419a4068d71 (127.4.5.65:35219)
I20240503 02:11:35.270596 5403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.65:35899
W20240503 02:11:35.636878 5595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:35.637287 5595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:35.637444 5595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:35.645669 5595 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:35.645862 5595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:35.646060 5595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.66
I20240503 02:11:35.656124 5595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.5.66
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.66
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:35.657950 5595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:35.660166 5595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:35.674878 5603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:35.675364 5601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:35.676044 5600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:35.676570 5595 server_base.cc:1034] running on GCE node
I20240503 02:11:35.677425 5595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:35.678768 5595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:35.679976 5595 hybrid_clock.cc:648] HybridClock initialized: now 1714702295679901 us; error 82 us; skew 500 ppm
I20240503 02:11:35.680341 5595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:35.683281 5595 webserver.cc:460] Webserver started at http://127.4.5.66:43703/ using document root <none> and password file <none>
I20240503 02:11:35.684237 5595 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:35.684361 5595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:35.684782 5595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:35.688025 5595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "6e4c8822478b44fba80bbe71f7be1bb0"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-zv3f"
I20240503 02:11:35.689075 5595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "6e4c8822478b44fba80bbe71f7be1bb0"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-zv3f"
I20240503 02:11:35.695665 5595 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20240503 02:11:35.700034 5609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.701892 5595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:35.702111 5595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "6e4c8822478b44fba80bbe71f7be1bb0"
format_stamp: "Formatted at 2024-05-03 02:11:35 on dist-test-slave-zv3f"
I20240503 02:11:35.702381 5595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:35.725975 5595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:35.727136 5595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:35.727478 5595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:35.728717 5595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:35.730616 5595 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:35.730731 5595 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.730859 5595 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:35.730935 5595 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:35.767102 5595 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.66:40933
I20240503 02:11:35.767151 5722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.66:40933 every 8 connection(s)
I20240503 02:11:35.768774 5595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:35.771651 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 5595
I20240503 02:11:35.771903 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:35.775585 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.67:0
--local_ip_for_outbound_sockets=127.4.5.67
--webserver_interface=127.4.5.67
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--builtin_ntp_servers=127.4.5.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:35.787047 5723 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35993
I20240503 02:11:35.787422 5723 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:35.788425 5723 heartbeater.cc:511] Master 127.4.5.126:35993 requested a full tablet report, sending...
I20240503 02:11:35.790203 5403 ts_manager.cc:194] Registered new tserver with Master: 6e4c8822478b44fba80bbe71f7be1bb0 (127.4.5.66:40933)
I20240503 02:11:35.791121 5403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.66:33307
W20240503 02:11:36.180817 5727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:36.181212 5727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:36.181375 5727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:36.189507 5727 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:36.189675 5727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:36.189874 5727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.67
I20240503 02:11:36.199605 5727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.5.67
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.67
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:36.201246 5727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:36.203310 5727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:36.217073 5732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:36.217479 5735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:36.217975 5733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:36.218633 5727 server_base.cc:1034] running on GCE node
I20240503 02:11:36.219293 5727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:36.220402 5727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:36.221640 5727 hybrid_clock.cc:648] HybridClock initialized: now 1714702296221586 us; error 43 us; skew 500 ppm
I20240503 02:11:36.221990 5727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:36.224560 5727 webserver.cc:460] Webserver started at http://127.4.5.67:44721/ using document root <none> and password file <none>
I20240503 02:11:36.225383 5727 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:36.225507 5727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:36.225881 5727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:36.231127 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "b21c24fce2c24088b7d745cf625017e7"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-zv3f"
I20240503 02:11:36.232023 5727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "b21c24fce2c24088b7d745cf625017e7"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-zv3f"
I20240503 02:11:36.237636 5727 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20240503 02:11:36.241129 5741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.242487 5727 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:36.242652 5727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "b21c24fce2c24088b7d745cf625017e7"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-zv3f"
I20240503 02:11:36.242888 5727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:36.263494 5727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:36.264501 5727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:36.264809 5727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:36.265882 5727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:36.267577 5727 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:36.267681 5727 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.267807 5727 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:36.267899 5727 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.274022 5591 heartbeater.cc:503] Master 127.4.5.126:35993 was elected leader, sending a full tablet report...
I20240503 02:11:36.302886 5727 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.67:36035
I20240503 02:11:36.302932 5854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.67:36035 every 8 connection(s)
I20240503 02:11:36.304483 5727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:36.310532 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 5727
I20240503 02:11:36.310739 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:36.315366 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.68:0
--local_ip_for_outbound_sockets=127.4.5.68
--webserver_interface=127.4.5.68
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--builtin_ntp_servers=127.4.5.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:36.322541 5855 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35993
I20240503 02:11:36.322906 5855 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:36.323887 5855 heartbeater.cc:511] Master 127.4.5.126:35993 requested a full tablet report, sending...
I20240503 02:11:36.325544 5403 ts_manager.cc:194] Registered new tserver with Master: b21c24fce2c24088b7d745cf625017e7 (127.4.5.67:36035)
I20240503 02:11:36.326467 5403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.67:36903
W20240503 02:11:36.698781 5859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:36.699183 5859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:36.699337 5859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:36.707309 5859 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:36.707480 5859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:36.707652 5859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.68
I20240503 02:11:36.717619 5859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.4.5.68
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.68
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:36.719373 5859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:36.721478 5859 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:36.735731 5865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:36.736251 5867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:36.736835 5864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:36.737427 5859 server_base.cc:1034] running on GCE node
I20240503 02:11:36.738160 5859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:36.739311 5859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:36.740543 5859 hybrid_clock.cc:648] HybridClock initialized: now 1714702296740483 us; error 54 us; skew 500 ppm
I20240503 02:11:36.740955 5859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:36.743597 5859 webserver.cc:460] Webserver started at http://127.4.5.68:44011/ using document root <none> and password file <none>
I20240503 02:11:36.744513 5859 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:36.744629 5859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:36.744967 5859 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:36.748179 5859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "67af772135234d4a8895fd4a9718cff5"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-zv3f"
I20240503 02:11:36.749110 5859 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "67af772135234d4a8895fd4a9718cff5"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-zv3f"
I20240503 02:11:36.755343 5859 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 02:11:36.759490 5873 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.761142 5859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:11:36.761296 5859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "67af772135234d4a8895fd4a9718cff5"
format_stamp: "Formatted at 2024-05-03 02:11:36 on dist-test-slave-zv3f"
I20240503 02:11:36.761494 5859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:36.786605 5859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:36.787635 5859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:36.787971 5859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:36.789113 5859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:36.790843 5859 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:36.790932 5859 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.791023 5859 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:36.791081 5859 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:36.794355 5723 heartbeater.cc:503] Master 127.4.5.126:35993 was elected leader, sending a full tablet report...
I20240503 02:11:36.827311 5859 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.68:33191
I20240503 02:11:36.827351 5987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.68:33191 every 8 connection(s)
I20240503 02:11:36.829293 5859 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:36.839319 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 5859
I20240503 02:11:36.839572 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:36.843987 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.69:0
--local_ip_for_outbound_sockets=127.4.5.69
--webserver_interface=127.4.5.69
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--builtin_ntp_servers=127.4.5.84:35873
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:36.846392 5988 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35993
I20240503 02:11:36.846805 5988 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:36.848016 5988 heartbeater.cc:511] Master 127.4.5.126:35993 requested a full tablet report, sending...
I20240503 02:11:36.849915 5403 ts_manager.cc:194] Registered new tserver with Master: 67af772135234d4a8895fd4a9718cff5 (127.4.5.68:33191)
I20240503 02:11:36.851260 5403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.68:59017
W20240503 02:11:37.242606 5992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:37.243045 5992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:37.243219 5992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:37.251698 5992 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:37.251854 5992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:37.252120 5992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.69
I20240503 02:11:37.262533 5992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:35873
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.4.5.69
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35993
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.69
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:37.264245 5992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:37.266500 5992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:37.281232 5998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:37.281740 5997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:37.282259 6000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:37.282999 5992 server_base.cc:1034] running on GCE node
I20240503 02:11:37.283756 5992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:37.285032 5992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:37.286254 5992 hybrid_clock.cc:648] HybridClock initialized: now 1714702297286195 us; error 57 us; skew 500 ppm
I20240503 02:11:37.286624 5992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:37.289314 5992 webserver.cc:460] Webserver started at http://127.4.5.69:45441/ using document root <none> and password file <none>
I20240503 02:11:37.290220 5992 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:37.290349 5992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:37.290706 5992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:37.293933 5992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "a01375bc370e486a9188d42cd939a2e9"
format_stamp: "Formatted at 2024-05-03 02:11:37 on dist-test-slave-zv3f"
I20240503 02:11:37.294852 5992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "a01375bc370e486a9188d42cd939a2e9"
format_stamp: "Formatted at 2024-05-03 02:11:37 on dist-test-slave-zv3f"
I20240503 02:11:37.300954 5992 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 02:11:37.304932 6006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:37.307276 5992 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:11:37.307466 5992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "a01375bc370e486a9188d42cd939a2e9"
format_stamp: "Formatted at 2024-05-03 02:11:37 on dist-test-slave-zv3f"
I20240503 02:11:37.307736 5992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:37.329520 5855 heartbeater.cc:503] Master 127.4.5.126:35993 was elected leader, sending a full tablet report...
I20240503 02:11:37.355095 5992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:37.356228 5992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:37.356583 5992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:37.357796 5992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:37.359725 5992 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:37.359828 5992 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:37.359966 5992 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:37.360044 5992 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:37.396108 5992 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.69:44657
I20240503 02:11:37.396159 6119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.69:44657 every 8 connection(s)
I20240503 02:11:37.397815 5992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:37.399130 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 5992
I20240503 02:11:37.399350 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceStartCriteriaTest.TabletServerIsDown_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:37.412977 6120 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35993
I20240503 02:11:37.413332 6120 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:37.414307 6120 heartbeater.cc:511] Master 127.4.5.126:35993 requested a full tablet report, sending...
I20240503 02:11:37.415967 5403 ts_manager.cc:194] Registered new tserver with Master: a01375bc370e486a9188d42cd939a2e9 (127.4.5.69:44657)
I20240503 02:11:37.417033 5403 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.69:40079
I20240503 02:11:37.417151 4117 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:37.434556 5403 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51758:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:11:37.489312 5658 tablet_service.cc:1467] Processing CreateTablet for tablet 38bbd02f34364ff6a76f6fad3fc08f9f (DEFAULT_TABLE table=TestTable [id=74c36fc68cb84b50a61fad5d0cef13f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.491241 5658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bbd02f34364ff6a76f6fad3fc08f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.490767 5922 tablet_service.cc:1467] Processing CreateTablet for tablet 38bbd02f34364ff6a76f6fad3fc08f9f (DEFAULT_TABLE table=TestTable [id=74c36fc68cb84b50a61fad5d0cef13f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.492354 5922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bbd02f34364ff6a76f6fad3fc08f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.492094 5525 tablet_service.cc:1467] Processing CreateTablet for tablet 38bbd02f34364ff6a76f6fad3fc08f9f (DEFAULT_TABLE table=TestTable [id=74c36fc68cb84b50a61fad5d0cef13f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:37.493670 5525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38bbd02f34364ff6a76f6fad3fc08f9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:37.506054 6139 tablet_bootstrap.cc:492] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71: Bootstrap starting.
I20240503 02:11:37.510064 6139 tablet_bootstrap.cc:654] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.511322 6139 log.cc:824] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:37.512528 6141 tablet_bootstrap.cc:492] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0: Bootstrap starting.
I20240503 02:11:37.516057 6140 tablet_bootstrap.cc:492] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5: Bootstrap starting.
I20240503 02:11:37.517637 6141 tablet_bootstrap.cc:654] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.518826 6141 log.cc:824] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:37.519996 6139 tablet_bootstrap.cc:492] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71: No bootstrap required, opened a new log
I20240503 02:11:37.520076 6140 tablet_bootstrap.cc:654] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:37.520364 6139 ts_tablet_manager.cc:1367] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240503 02:11:37.521533 6140 log.cc:824] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:37.522162 6141 tablet_bootstrap.cc:492] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0: No bootstrap required, opened a new log
I20240503 02:11:37.522532 6141 ts_tablet_manager.cc:1367] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.007s
I20240503 02:11:37.529240 6141 raft_consensus.cc:348] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.529800 6141 raft_consensus.cc:374] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.529932 6141 raft_consensus.cc:729] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e4c8822478b44fba80bbe71f7be1bb0, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.530789 6141 consensus_queue.cc:260] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.531744 6139 raft_consensus.cc:348] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.532245 6139 raft_consensus.cc:374] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.532370 6139 raft_consensus.cc:729] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3892a5f8dbc4c05bea4f419a4068d71, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.533174 6141 ts_tablet_manager.cc:1398] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
I20240503 02:11:37.533162 6139 consensus_queue.cc:260] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.531824 6140 tablet_bootstrap.cc:492] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5: No bootstrap required, opened a new log
I20240503 02:11:37.533886 6140 ts_tablet_manager.cc:1367] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5: Time spent bootstrapping tablet: real 0.018s user 0.016s sys 0.000s
I20240503 02:11:37.536880 6139 ts_tablet_manager.cc:1398] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71: Time spent starting tablet: real 0.016s user 0.012s sys 0.002s
I20240503 02:11:37.541973 6140 raft_consensus.cc:348] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.542438 6140 raft_consensus.cc:374] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:37.542557 6140 raft_consensus.cc:729] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67af772135234d4a8895fd4a9718cff5, State: Initialized, Role: FOLLOWER
I20240503 02:11:37.543314 6140 consensus_queue.cc:260] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.544490 5988 heartbeater.cc:503] Master 127.4.5.126:35993 was elected leader, sending a full tablet report...
I20240503 02:11:37.545159 6140 ts_tablet_manager.cc:1398] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
W20240503 02:11:37.582077 5989 tablet.cc:2369] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:11:37.752177 5592 tablet.cc:2369] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:37.757941 6147 raft_consensus.cc:482] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:11:37.758273 6147 raft_consensus.cc:504] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.760531 6147 leader_election.cc:288] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e4c8822478b44fba80bbe71f7be1bb0 (127.4.5.66:40933), f3892a5f8dbc4c05bea4f419a4068d71 (127.4.5.65:35219)
I20240503 02:11:37.769356 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bbd02f34364ff6a76f6fad3fc08f9f" candidate_uuid: "67af772135234d4a8895fd4a9718cff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" is_pre_election: true
I20240503 02:11:37.769385 5545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bbd02f34364ff6a76f6fad3fc08f9f" candidate_uuid: "67af772135234d4a8895fd4a9718cff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" is_pre_election: true
I20240503 02:11:37.770038 5678 raft_consensus.cc:2454] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67af772135234d4a8895fd4a9718cff5 in term 0.
I20240503 02:11:37.770043 5545 raft_consensus.cc:2454] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 67af772135234d4a8895fd4a9718cff5 in term 0.
I20240503 02:11:37.770848 5878 leader_election.cc:302] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67af772135234d4a8895fd4a9718cff5, f3892a5f8dbc4c05bea4f419a4068d71; no voters:
I20240503 02:11:37.771358 6147 raft_consensus.cc:2781] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:11:37.771544 6147 raft_consensus.cc:482] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:11:37.771657 6147 raft_consensus.cc:3037] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:11:37.771965 5724 tablet.cc:2369] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:11:37.775152 6147 raft_consensus.cc:504] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.776074 6147 leader_election.cc:288] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [CANDIDATE]: Term 1 election: Requested vote from peers 6e4c8822478b44fba80bbe71f7be1bb0 (127.4.5.66:40933), f3892a5f8dbc4c05bea4f419a4068d71 (127.4.5.65:35219)
I20240503 02:11:37.776782 5678 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bbd02f34364ff6a76f6fad3fc08f9f" candidate_uuid: "67af772135234d4a8895fd4a9718cff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e4c8822478b44fba80bbe71f7be1bb0"
I20240503 02:11:37.776808 5545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38bbd02f34364ff6a76f6fad3fc08f9f" candidate_uuid: "67af772135234d4a8895fd4a9718cff5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3892a5f8dbc4c05bea4f419a4068d71"
I20240503 02:11:37.777091 5545 raft_consensus.cc:3037] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.777084 5678 raft_consensus.cc:3037] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:37.780215 5545 raft_consensus.cc:2454] T 38bbd02f34364ff6a76f6fad3fc08f9f P f3892a5f8dbc4c05bea4f419a4068d71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67af772135234d4a8895fd4a9718cff5 in term 1.
I20240503 02:11:37.780591 5678 raft_consensus.cc:2454] T 38bbd02f34364ff6a76f6fad3fc08f9f P 6e4c8822478b44fba80bbe71f7be1bb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 67af772135234d4a8895fd4a9718cff5 in term 1.
I20240503 02:11:37.780872 5878 leader_election.cc:302] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67af772135234d4a8895fd4a9718cff5, f3892a5f8dbc4c05bea4f419a4068d71; no voters:
I20240503 02:11:37.781459 6147 raft_consensus.cc:2781] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:37.782008 6147 raft_consensus.cc:686] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [term 1 LEADER]: Becoming Leader. State: Replica: 67af772135234d4a8895fd4a9718cff5, State: Running, Role: LEADER
I20240503 02:11:37.782618 6147 consensus_queue.cc:237] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } }
I20240503 02:11:37.789201 5403 catalog_manager.cc:5484] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 67af772135234d4a8895fd4a9718cff5 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "67af772135234d4a8895fd4a9718cff5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67af772135234d4a8895fd4a9718cff5" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 33191 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f3892a5f8dbc4c05bea4f419a4068d71" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 35219 } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:37.858001 4117 external_mini_cluster.cc:930] 5 TS(s) registered with all masters
I20240503 02:11:37.860574 4117 ts_itest-base.cc:233] Waiting for 1 tablets on tserver f3892a5f8dbc4c05bea4f419a4068d71 to finish bootstrapping
I20240503 02:11:37.871022 4117 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6e4c8822478b44fba80bbe71f7be1bb0 to finish bootstrapping
I20240503 02:11:37.880553 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver b21c24fce2c24088b7d745cf625017e7 to finish bootstrapping
I20240503 02:11:37.891511 4117 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 67af772135234d4a8895fd4a9718cff5 to finish bootstrapping
I20240503 02:11:37.900800 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver a01375bc370e486a9188d42cd939a2e9 to finish bootstrapping
I20240503 02:11:37.911617 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 5462
I20240503 02:11:37.912904 5586 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:38.007315 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 5462
I20240503 02:11:38.282925 6147 consensus_queue.cc:1035] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e4c8822478b44fba80bbe71f7be1bb0" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 40933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:11:38.299012 5878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.65:35219: connect: Connection refused (error 111)
W20240503 02:11:38.306007 5878 consensus_peers.cc:485] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 -> Peer f3892a5f8dbc4c05bea4f419a4068d71 (127.4.5.65:35219): Couldn't send request to peer f3892a5f8dbc4c05bea4f419a4068d71. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.65:35219: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:38.419924 6120 heartbeater.cc:503] Master 127.4.5.126:35993 was elected leader, sending a full tablet report...
I20240503 02:11:38.580436 5658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.582178 5922 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:38.584024 5789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.595371 6055 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.753186 6055 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.754354 5658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.755450 5922 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:38.759447 5789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.952709 5922 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:38.957788 6055 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.957924 5789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:38.962867 5658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:39.700948 5922 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:39.701026 5658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:39.704114 6055 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:39.708487 5789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:39.857955 5922 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:39.863240 5658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:39.866030 6055 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:39.869745 5789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:40.061244 5922 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:40.073098 5789 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:40.073307 5658 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:40.092747 6055 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:40.255601 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 5595
I20240503 02:11:40.257536 5718 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:40.375902 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 5595
W20240503 02:11:40.396687 5877 connection.cc:535] client connection to 127.4.5.66:40933 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:11:40.397150 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 5727
W20240503 02:11:40.397231 5390 connection.cc:535] server connection from 127.4.5.66:33307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:11:40.398365 5877 consensus_peers.cc:485] T 38bbd02f34364ff6a76f6fad3fc08f9f P 67af772135234d4a8895fd4a9718cff5 -> Peer 6e4c8822478b44fba80bbe71f7be1bb0 (127.4.5.66:40933): Couldn't send request to peer 6e4c8822478b44fba80bbe71f7be1bb0. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.66:40933: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:40.398506 5849 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:40.511442 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 5727
I20240503 02:11:40.530928 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 5859
I20240503 02:11:40.532147 5982 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:40.651106 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 5859
I20240503 02:11:40.671190 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 5992
I20240503 02:11:40.672537 6115 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:40.778738 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 5992
I20240503 02:11:40.797880 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 5372
I20240503 02:11:40.799275 5433 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:40.940279 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 5372
2024-05-03T02:11:40Z chronyd exiting
[ OK ] RebalanceStartCriteriaTest.TabletServerIsDown/1 (6912 ms)
[----------] 1 test from RebalanceStartCriteriaTest (6912 ms total)
[----------] 1 test from RebalanceParamTest
[ RUN ] RebalanceParamTest.Rebalance/1
I20240503 02:11:40.989444 4117 test_util.cc:268] Using random seed: 1032570842
I20240503 02:11:40.991645 4117 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:11:40.991730 4117 ts_itest-base.cc:113] --------------
I20240503 02:11:40.991802 4117 ts_itest-base.cc:114] 7 tablet servers
I20240503 02:11:40.991861 4117 ts_itest-base.cc:115] 1 replicas per TS
I20240503 02:11:40.991933 4117 ts_itest-base.cc:116] --------------
2024-05-03T02:11:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:11:41Z Disabled control of system clock
I20240503 02:11:41.022145 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.5.126:42813
--webserver_interface=127.4.5.126
--webserver_port=0
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.5.126:42813
--allow_unsafe_replication_factor
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--tserver_unresponsive_timeout_ms=3000 with env {}
W20240503 02:11:41.437208 6373 flags.cc:425] Enabled unsafe flag: --allow_unsafe_replication_factor=true
W20240503 02:11:41.437641 6373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:41.437726 6373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:41.437809 6373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:41.446368 6373 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:41.446523 6373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:11:41.446592 6373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:41.446650 6373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:11:41.446745 6373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:11:41.457849 6373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
--allow_unsafe_replication_factor=true
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.4.5.126:42813
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.5.126:42813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.5.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:41.459630 6373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:41.461759 6373 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:41.474690 6379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.475306 6378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:41.475883 6381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:41.476552 6373 server_base.cc:1034] running on GCE node
I20240503 02:11:41.477624 6373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:41.479530 6373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:41.480753 6373 hybrid_clock.cc:648] HybridClock initialized: now 1714702301480669 us; error 64 us; skew 500 ppm
I20240503 02:11:41.481178 6373 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:41.484210 6373 webserver.cc:460] Webserver started at http://127.4.5.126:40259/ using document root <none> and password file <none>
I20240503 02:11:41.485168 6373 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:41.485322 6373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:41.485700 6373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:41.489096 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "596ca00a45524913952aa8c8b8864eee"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-zv3f"
I20240503 02:11:41.490056 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "596ca00a45524913952aa8c8b8864eee"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-zv3f"
I20240503 02:11:41.496716 6373 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 02:11:41.501196 6387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:41.502910 6373 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:11:41.503118 6373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
uuid: "596ca00a45524913952aa8c8b8864eee"
format_stamp: "Formatted at 2024-05-03 02:11:41 on dist-test-slave-zv3f"
I20240503 02:11:41.503355 6373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:41.524041 6373 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:41.525303 6373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:41.525679 6373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:41.547195 6373 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.126:42813
I20240503 02:11:41.547216 6438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.126:42813 every 8 connection(s)
I20240503 02:11:41.549054 6373 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:11:41.553913 6439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:41.556188 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 6373
I20240503 02:11:41.556461 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:11:41.567292 6439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee: Bootstrap starting.
I20240503 02:11:41.572507 6439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:41.574187 6439 log.cc:824] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:41.577811 6439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee: No bootstrap required, opened a new log
I20240503 02:11:41.585095 6439 raft_consensus.cc:348] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596ca00a45524913952aa8c8b8864eee" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42813 } }
I20240503 02:11:41.585661 6439 raft_consensus.cc:374] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:41.585804 6439 raft_consensus.cc:729] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 596ca00a45524913952aa8c8b8864eee, State: Initialized, Role: FOLLOWER
I20240503 02:11:41.586581 6439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596ca00a45524913952aa8c8b8864eee" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42813 } }
I20240503 02:11:41.586798 6439 raft_consensus.cc:388] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:41.586910 6439 raft_consensus.cc:482] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:41.587059 6439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:41.596818 6439 raft_consensus.cc:504] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596ca00a45524913952aa8c8b8864eee" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42813 } }
I20240503 02:11:41.597692 6439 leader_election.cc:302] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 596ca00a45524913952aa8c8b8864eee; no voters:
I20240503 02:11:41.598495 6439 leader_election.cc:288] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:41.598698 6445 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:41.600116 6445 raft_consensus.cc:686] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [term 1 LEADER]: Becoming Leader. State: Replica: 596ca00a45524913952aa8c8b8864eee, State: Running, Role: LEADER
I20240503 02:11:41.601007 6445 consensus_queue.cc:237] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596ca00a45524913952aa8c8b8864eee" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42813 } }
I20240503 02:11:41.601284 6439 sys_catalog.cc:564] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:11:41.604971 6447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [sys.catalog]: SysCatalogTable state changed. Reason: New leader 596ca00a45524913952aa8c8b8864eee. Latest consensus state: current_term: 1 leader_uuid: "596ca00a45524913952aa8c8b8864eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596ca00a45524913952aa8c8b8864eee" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42813 } } }
I20240503 02:11:41.605044 6446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "596ca00a45524913952aa8c8b8864eee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "596ca00a45524913952aa8c8b8864eee" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 42813 } } }
I20240503 02:11:41.605368 6447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:41.605448 6446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee [sys.catalog]: This master's current role is: LEADER
I20240503 02:11:41.606493 6454 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:11:41.615281 6454 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:11:41.628993 6454 catalog_manager.cc:1347] Generated new cluster ID: 31eb16ad207d431daa00e068e2e1abde
I20240503 02:11:41.629199 6454 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:11:41.659605 6454 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:11:41.661967 6454 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:11:41.682790 6454 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 596ca00a45524913952aa8c8b8864eee: Generated new TSK 0
I20240503 02:11:41.683986 6454 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:11:41.702848 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.65:0
--local_ip_for_outbound_sockets=127.4.5.65
--webserver_interface=127.4.5.65
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:42.135295 6464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:42.135737 6464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:42.135854 6464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:42.144529 6464 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:42.144732 6464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:42.144881 6464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.65
I20240503 02:11:42.155426 6464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.5.65
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.65
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:42.157327 6464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:42.159616 6464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.174995 6469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.175513 6470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.176126 6472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.176853 6464 server_base.cc:1034] running on GCE node
I20240503 02:11:42.177659 6464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:42.178941 6464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:42.180176 6464 hybrid_clock.cc:648] HybridClock initialized: now 1714702302180103 us; error 59 us; skew 500 ppm
I20240503 02:11:42.180673 6464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.183697 6464 webserver.cc:460] Webserver started at http://127.4.5.65:36791/ using document root <none> and password file <none>
I20240503 02:11:42.184748 6464 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.184901 6464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.185305 6464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:42.188758 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "d305ba2a2a4640f7997613a7b8884dee"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-zv3f"
I20240503 02:11:42.189733 6464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "d305ba2a2a4640f7997613a7b8884dee"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-zv3f"
I20240503 02:11:42.196396 6464 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20240503 02:11:42.200845 6478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.202785 6464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:11:42.202998 6464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "d305ba2a2a4640f7997613a7b8884dee"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-zv3f"
I20240503 02:11:42.203251 6464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.226912 6464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.228116 6464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:42.228497 6464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.229758 6464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.231833 6464 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.231935 6464 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.232066 6464 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.232162 6464 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.271252 6464 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.65:39365
I20240503 02:11:42.271291 6590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.65:39365 every 8 connection(s)
I20240503 02:11:42.273069 6464 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:11:42.283396 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 6464
I20240503 02:11:42.283699 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:11:42.287909 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.66:0
--local_ip_for_outbound_sockets=127.4.5.66
--webserver_interface=127.4.5.66
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:42.289192 6591 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:42.289649 6591 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.290777 6591 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:42.295871 6404 ts_manager.cc:194] Registered new tserver with Master: d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:11:42.297866 6404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.65:38947
W20240503 02:11:42.697204 6595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:42.697600 6595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:42.697747 6595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:42.706264 6595 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:42.706429 6595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:42.706619 6595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.66
I20240503 02:11:42.717254 6595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.5.66
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.66
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:42.719024 6595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:42.721185 6595 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:42.736536 6601 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.736932 6600 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:42.737522 6603 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:42.738338 6595 server_base.cc:1034] running on GCE node
I20240503 02:11:42.739094 6595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:42.740309 6595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:42.741537 6595 hybrid_clock.cc:648] HybridClock initialized: now 1714702302741481 us; error 53 us; skew 500 ppm
I20240503 02:11:42.741921 6595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:42.744897 6595 webserver.cc:460] Webserver started at http://127.4.5.66:44019/ using document root <none> and password file <none>
I20240503 02:11:42.745837 6595 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:42.745982 6595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:42.746378 6595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:42.749698 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "4e21bb5794994a489ebd09ee30b6df70"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-zv3f"
I20240503 02:11:42.750656 6595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "4e21bb5794994a489ebd09ee30b6df70"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-zv3f"
I20240503 02:11:42.757102 6595 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 02:11:42.761482 6609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.763257 6595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:11:42.763468 6595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "4e21bb5794994a489ebd09ee30b6df70"
format_stamp: "Formatted at 2024-05-03 02:11:42 on dist-test-slave-zv3f"
I20240503 02:11:42.763718 6595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:42.783383 6595 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:42.784590 6595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:42.784933 6595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:42.786080 6595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:42.788031 6595 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:42.788147 6595 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.788283 6595 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:42.788362 6595 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:42.827507 6595 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.66:39573
I20240503 02:11:42.827569 6721 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.66:39573 every 8 connection(s)
I20240503 02:11:42.829401 6595 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:11:42.836185 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 6595
I20240503 02:11:42.836378 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:11:42.840266 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.67:0
--local_ip_for_outbound_sockets=127.4.5.67
--webserver_interface=127.4.5.67
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:42.848279 6722 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:42.848774 6722 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:42.849900 6722 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:42.852026 6404 ts_manager.cc:194] Registered new tserver with Master: 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:11:42.853469 6404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.66:40977
W20240503 02:11:43.254961 6727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:43.255388 6727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:43.255517 6727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:43.263864 6727 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:43.264048 6727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:43.264200 6727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.67
I20240503 02:11:43.274860 6727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.5.67
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.67
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:43.276903 6727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:43.279171 6727 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:43.293952 6735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.294561 6733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.295208 6732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:43.295981 6727 server_base.cc:1034] running on GCE node
I20240503 02:11:43.296818 6727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:43.298019 6727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:43.299228 6727 hybrid_clock.cc:648] HybridClock initialized: now 1714702303299162 us; error 62 us; skew 500 ppm
I20240503 02:11:43.299603 6727 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:43.301971 6591 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:43.302786 6727 webserver.cc:460] Webserver started at http://127.4.5.67:38591/ using document root <none> and password file <none>
I20240503 02:11:43.303716 6727 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:43.303856 6727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:43.304313 6727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:43.307567 6727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "5ad3a6abdbc846c1a6ed42269a100dd3"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-zv3f"
I20240503 02:11:43.308531 6727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "5ad3a6abdbc846c1a6ed42269a100dd3"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-zv3f"
I20240503 02:11:43.314895 6727 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 02:11:43.319087 6741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.320922 6727 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20240503 02:11:43.321142 6727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5ad3a6abdbc846c1a6ed42269a100dd3"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-zv3f"
I20240503 02:11:43.321414 6727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:43.359987 6727 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:43.361294 6727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:43.361665 6727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:43.362957 6727 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:43.364877 6727 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:43.364996 6727 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.365129 6727 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:43.365211 6727 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.402849 6727 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.67:41365
I20240503 02:11:43.402918 6855 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.67:41365 every 8 connection(s)
I20240503 02:11:43.404582 6727 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:43.407915 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 6727
I20240503 02:11:43.408331 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:11:43.412294 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.68:0
--local_ip_for_outbound_sockets=127.4.5.68
--webserver_interface=127.4.5.68
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:43.421800 6856 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:43.422268 6856 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:43.423664 6856 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:43.425871 6404 ts_manager.cc:194] Registered new tserver with Master: 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67:41365)
I20240503 02:11:43.426898 6404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.67:35119
W20240503 02:11:43.833803 6860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:43.834213 6860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:43.834393 6860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:43.843847 6860 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:43.844095 6860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:43.844291 6860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.68
I20240503 02:11:43.855069 6860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.4.5.68
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.68
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:43.857091 6860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:43.857319 6722 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:43.859712 6860 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:43.875917 6866 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.876560 6868 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:43.877215 6865 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:43.877965 6860 server_base.cc:1034] running on GCE node
I20240503 02:11:43.878772 6860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:43.880019 6860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:43.881281 6860 hybrid_clock.cc:648] HybridClock initialized: now 1714702303881191 us; error 66 us; skew 500 ppm
I20240503 02:11:43.881696 6860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:43.884637 6860 webserver.cc:460] Webserver started at http://127.4.5.68:33247/ using document root <none> and password file <none>
I20240503 02:11:43.885535 6860 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:43.885679 6860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:43.886044 6860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:43.889163 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "6f857824c6df4cb3a0eb69a4b66810f9"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-zv3f"
I20240503 02:11:43.890048 6860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "6f857824c6df4cb3a0eb69a4b66810f9"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-zv3f"
I20240503 02:11:43.896029 6860 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240503 02:11:43.899930 6874 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.901572 6860 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20240503 02:11:43.901746 6860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6f857824c6df4cb3a0eb69a4b66810f9"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-zv3f"
I20240503 02:11:43.901988 6860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:43.925710 6860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:43.926934 6860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:43.927347 6860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:43.928689 6860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:43.930568 6860 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:43.930678 6860 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.930820 6860 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:43.930897 6860 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:43.970063 6860 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.68:42071
I20240503 02:11:43.970122 6987 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.68:42071 every 8 connection(s)
I20240503 02:11:43.971686 6860 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:43.980054 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 6860
I20240503 02:11:43.980299 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:11:43.984606 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.69:0
--local_ip_for_outbound_sockets=127.4.5.69
--webserver_interface=127.4.5.69
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:43.988018 6988 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:43.988435 6988 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:43.989681 6988 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:43.991684 6404 ts_manager.cc:194] Registered new tserver with Master: 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68:42071)
I20240503 02:11:43.992678 6404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.68:47329
W20240503 02:11:44.385516 6992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:44.385973 6992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:44.386158 6992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:44.394873 6992 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:44.395090 6992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:44.395293 6992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.69
I20240503 02:11:44.408067 6992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.4.5.69
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.69
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:44.410188 6992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:44.412634 6992 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:44.427688 6998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:44.428205 6997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:44.428855 7000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:44.429687 6992 server_base.cc:1034] running on GCE node
I20240503 02:11:44.430488 6992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:44.431499 6856 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:44.431764 6992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:44.432998 6992 hybrid_clock.cc:648] HybridClock initialized: now 1714702304432929 us; error 63 us; skew 500 ppm
I20240503 02:11:44.433444 6992 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:44.436230 6992 webserver.cc:460] Webserver started at http://127.4.5.69:37647/ using document root <none> and password file <none>
I20240503 02:11:44.437353 6992 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:44.437495 6992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:44.437862 6992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:44.441238 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e3fafd8e3e974740b2525f2e89d7cca5"
format_stamp: "Formatted at 2024-05-03 02:11:44 on dist-test-slave-zv3f"
I20240503 02:11:44.442234 6992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e3fafd8e3e974740b2525f2e89d7cca5"
format_stamp: "Formatted at 2024-05-03 02:11:44 on dist-test-slave-zv3f"
I20240503 02:11:44.449191 6992 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20240503 02:11:44.453425 7006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:44.455103 6992 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.005s sys 0.000s
I20240503 02:11:44.455304 6992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e3fafd8e3e974740b2525f2e89d7cca5"
format_stamp: "Formatted at 2024-05-03 02:11:44 on dist-test-slave-zv3f"
I20240503 02:11:44.455550 6992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:44.474874 6992 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:44.475975 6992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:44.476362 6992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:44.477571 6992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:44.479280 6992 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:44.479372 6992 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:44.479471 6992 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:44.479530 6992 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:44.514768 6992 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.69:35867
I20240503 02:11:44.514815 7119 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.69:35867 every 8 connection(s)
I20240503 02:11:44.516371 6992 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:44.520138 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 6992
I20240503 02:11:44.520365 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:11:44.524127 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.70:0
--local_ip_for_outbound_sockets=127.4.5.70
--webserver_interface=127.4.5.70
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:44.532279 7120 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:44.532734 7120 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:44.534009 7120 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:44.535972 6404 ts_manager.cc:194] Registered new tserver with Master: e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69:35867)
I20240503 02:11:44.536952 6404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.69:51137
W20240503 02:11:44.923499 7124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:44.923885 7124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:44.924031 7124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:44.932420 7124 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:44.932644 7124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:44.932860 7124 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.70
I20240503 02:11:44.943045 7124 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.4.5.70
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.70
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:44.944878 7124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:44.947101 7124 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:44.961789 7132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:44.962421 7130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:44.963164 7129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:44.963981 7124 server_base.cc:1034] running on GCE node
I20240503 02:11:44.964780 7124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:44.965996 7124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:44.967250 7124 hybrid_clock.cc:648] HybridClock initialized: now 1714702304967149 us; error 68 us; skew 500 ppm
I20240503 02:11:44.967648 7124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:44.976536 7124 webserver.cc:460] Webserver started at http://127.4.5.70:35955/ using document root <none> and password file <none>
I20240503 02:11:44.977499 7124 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:44.977632 7124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:44.978003 7124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:44.981202 7124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "329369f9124942438b3176e3b0bacf98"
format_stamp: "Formatted at 2024-05-03 02:11:44 on dist-test-slave-zv3f"
I20240503 02:11:44.982112 7124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "329369f9124942438b3176e3b0bacf98"
format_stamp: "Formatted at 2024-05-03 02:11:44 on dist-test-slave-zv3f"
I20240503 02:11:44.988600 7124 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20240503 02:11:44.993240 7138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:44.995113 7124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:44.995359 7124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "329369f9124942438b3176e3b0bacf98"
format_stamp: "Formatted at 2024-05-03 02:11:44 on dist-test-slave-zv3f"
I20240503 02:11:44.995663 7124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:44.996620 6988 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:45.017921 7124 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:45.019013 7124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:45.019341 7124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:45.020478 7124 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:45.022251 7124 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:45.022346 7124 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:45.022481 7124 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:45.022558 7124 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:45.069957 7124 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.70:40279
I20240503 02:11:45.070015 7252 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.70:40279 every 8 connection(s)
I20240503 02:11:45.071552 7124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:45.080384 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 7124
I20240503 02:11:45.080634 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:11:45.084216 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.71:0
--local_ip_for_outbound_sockets=127.4.5.71
--webserver_interface=127.4.5.71
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:45.087927 7253 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:45.088330 7253 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:45.089393 7253 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:45.091460 6404 ts_manager.cc:194] Registered new tserver with Master: 329369f9124942438b3176e3b0bacf98 (127.4.5.70:40279)
I20240503 02:11:45.092579 6404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.70:57929
W20240503 02:11:45.496220 7257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:45.496752 7257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:45.496937 7257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:45.506136 7257 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:45.506361 7257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:45.506557 7257 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.71
I20240503 02:11:45.519505 7257 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.4.5.71
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.71
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:45.521602 7257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:45.524008 7257 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:45.539465 7262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:45.540019 7263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:45.540843 7265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:45.540853 7120 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:45.541545 7257 server_base.cc:1034] running on GCE node
I20240503 02:11:45.542519 7257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:45.543938 7257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:45.545212 7257 hybrid_clock.cc:648] HybridClock initialized: now 1714702305545129 us; error 78 us; skew 500 ppm
I20240503 02:11:45.545668 7257 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:45.549146 7257 webserver.cc:460] Webserver started at http://127.4.5.71:33157/ using document root <none> and password file <none>
I20240503 02:11:45.550176 7257 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:45.550312 7257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:45.550724 7257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:11:45.554414 7257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "4bf9c9a4cde949dbabf5144188edbf5b"
format_stamp: "Formatted at 2024-05-03 02:11:45 on dist-test-slave-zv3f"
I20240503 02:11:45.555562 7257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "4bf9c9a4cde949dbabf5144188edbf5b"
format_stamp: "Formatted at 2024-05-03 02:11:45 on dist-test-slave-zv3f"
I20240503 02:11:45.563486 7257 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20240503 02:11:45.568593 7271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:45.570608 7257 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:11:45.570847 7257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4bf9c9a4cde949dbabf5144188edbf5b"
format_stamp: "Formatted at 2024-05-03 02:11:45 on dist-test-slave-zv3f"
I20240503 02:11:45.571105 7257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:45.602959 7257 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:45.604104 7257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:45.604490 7257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:45.605747 7257 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:45.607777 7257 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:45.607890 7257 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:45.608019 7257 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:45.608090 7257 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:45.646127 7257 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.71:37017
I20240503 02:11:45.646178 7385 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.71:37017 every 8 connection(s)
I20240503 02:11:45.648048 7257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:11:45.651252 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 7257
I20240503 02:11:45.651538 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:11:45.665673 7386 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:45.666081 7386 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:45.667173 7386 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:45.668977 6404 ts_manager.cc:194] Registered new tserver with Master: 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71:37017)
I20240503 02:11:45.670007 4117 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:11:45.670198 6404 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.71:48565
I20240503 02:11:45.689837 6404 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57688:
name: "TestTable"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
owner: "alice"
I20240503 02:11:45.724107 6923 tablet_service.cc:1467] Processing CreateTablet for tablet 83124ba50be244e3bcc43c41e9b644ec (DEFAULT_TABLE table=TestTable [id=7fa2b89d47764f5fbebb27a5680a1fb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:45.726186 6923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83124ba50be244e3bcc43c41e9b644ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:45.739331 7401 tablet_bootstrap.cc:492] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Bootstrap starting.
I20240503 02:11:45.743100 7401 tablet_bootstrap.cc:654] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:45.744423 7401 log.cc:824] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:45.747735 7401 tablet_bootstrap.cc:492] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: No bootstrap required, opened a new log
I20240503 02:11:45.748039 7401 ts_tablet_manager.cc:1367] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20240503 02:11:45.754878 7401 raft_consensus.cc:348] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } }
I20240503 02:11:45.755434 7401 raft_consensus.cc:374] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:45.755565 7401 raft_consensus.cc:729] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f857824c6df4cb3a0eb69a4b66810f9, State: Initialized, Role: FOLLOWER
I20240503 02:11:45.756323 7401 consensus_queue.cc:260] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } }
I20240503 02:11:45.756606 7401 raft_consensus.cc:388] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:45.756753 7401 raft_consensus.cc:482] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:45.756922 7401 raft_consensus.cc:3037] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:45.759204 7401 raft_consensus.cc:504] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } }
I20240503 02:11:45.759814 7401 leader_election.cc:302] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f857824c6df4cb3a0eb69a4b66810f9; no voters:
I20240503 02:11:45.760527 7401 leader_election.cc:288] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:45.760740 7403 raft_consensus.cc:2781] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:45.761521 7403 raft_consensus.cc:686] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 LEADER]: Becoming Leader. State: Replica: 6f857824c6df4cb3a0eb69a4b66810f9, State: Running, Role: LEADER
I20240503 02:11:45.762238 7403 consensus_queue.cc:237] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } }
I20240503 02:11:45.762681 7401 ts_tablet_manager.cc:1398] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20240503 02:11:45.768929 6404 catalog_manager.cc:5484] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:45.785446 4117 external_mini_cluster.cc:930] 7 TS(s) registered with all masters
I20240503 02:11:45.787743 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver d305ba2a2a4640f7997613a7b8884dee to finish bootstrapping
I20240503 02:11:45.799006 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4e21bb5794994a489ebd09ee30b6df70 to finish bootstrapping
I20240503 02:11:45.810047 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 5ad3a6abdbc846c1a6ed42269a100dd3 to finish bootstrapping
I20240503 02:11:45.821965 4117 ts_itest-base.cc:233] Waiting for 1 tablets on tserver 6f857824c6df4cb3a0eb69a4b66810f9 to finish bootstrapping
I20240503 02:11:45.832365 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver e3fafd8e3e974740b2525f2e89d7cca5 to finish bootstrapping
I20240503 02:11:45.843330 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 329369f9124942438b3176e3b0bacf98 to finish bootstrapping
I20240503 02:11:45.853564 4117 ts_itest-base.cc:233] Waiting for 0 tablets on tserver 4bf9c9a4cde949dbabf5144188edbf5b to finish bootstrapping
I20240503 02:11:45.864814 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 6727
I20240503 02:11:45.866055 6851 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:45.958779 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 6727
I20240503 02:11:45.979919 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 6860
I20240503 02:11:45.981397 6983 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:46.089900 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 6860
I20240503 02:11:46.096781 7253 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:46.108505 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 6992
I20240503 02:11:46.109998 7115 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:46.198475 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 6992
I20240503 02:11:46.216089 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 7124
I20240503 02:11:46.217459 7248 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:46.305845 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 7124
I20240503 02:11:46.325409 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 7257
I20240503 02:11:46.326743 7381 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:11:46.413349 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 7257
I20240503 02:11:50.183156 6404 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57688:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:50.206274 6657 tablet_service.cc:1467] Processing CreateTablet for tablet 8a3067767dbf425c8a2a662ec01aa935 (DEFAULT_TABLE table=rebalance_test_table_0 [id=598e8f31ad644c8fbbaceeffbce04724]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.206343 6526 tablet_service.cc:1467] Processing CreateTablet for tablet d81a727ce55b4e629da976812205fb46 (DEFAULT_TABLE table=rebalance_test_table_0 [id=598e8f31ad644c8fbbaceeffbce04724]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.206303 6525 tablet_service.cc:1467] Processing CreateTablet for tablet 587895f39d8b47858e704e8ccb274b89 (DEFAULT_TABLE table=rebalance_test_table_0 [id=598e8f31ad644c8fbbaceeffbce04724]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:50.207942 6657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8a3067767dbf425c8a2a662ec01aa935. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.208019 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587895f39d8b47858e704e8ccb274b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.208608 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d81a727ce55b4e629da976812205fb46. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:50.222779 7429 tablet_bootstrap.cc:492] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70: Bootstrap starting.
I20240503 02:11:50.223924 7430 tablet_bootstrap.cc:492] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee: Bootstrap starting.
I20240503 02:11:50.226379 7429 tablet_bootstrap.cc:654] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.226950 7430 tablet_bootstrap.cc:654] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.227656 7429 log.cc:824] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:50.228094 7430 log.cc:824] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:50.230973 7430 tablet_bootstrap.cc:492] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee: No bootstrap required, opened a new log
I20240503 02:11:50.230973 7429 tablet_bootstrap.cc:492] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70: No bootstrap required, opened a new log
I20240503 02:11:50.231254 7429 ts_tablet_manager.cc:1367] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:11:50.231252 7430 ts_tablet_manager.cc:1367] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 02:11:50.237855 7429 raft_consensus.cc:348] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:50.237854 7430 raft_consensus.cc:348] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:50.238301 7430 raft_consensus.cc:374] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:50.238303 7429 raft_consensus.cc:374] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:50.238422 7429 raft_consensus.cc:729] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Initialized, Role: FOLLOWER
I20240503 02:11:50.238473 7430 raft_consensus.cc:729] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Initialized, Role: FOLLOWER
I20240503 02:11:50.239228 7430 consensus_queue.cc:260] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:50.239209 7429 consensus_queue.cc:260] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:50.239440 7429 raft_consensus.cc:388] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:50.239444 7430 raft_consensus.cc:388] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:50.239581 7430 raft_consensus.cc:482] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:50.239585 7429 raft_consensus.cc:482] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:50.239738 7430 raft_consensus.cc:3037] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:50.239738 7429 raft_consensus.cc:3037] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:50.241954 7429 raft_consensus.cc:504] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:50.241971 7430 raft_consensus.cc:504] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:50.242483 7429 leader_election.cc:302] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:11:50.242484 7430 leader_election.cc:302] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:50.243255 7429 leader_election.cc:288] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:50.243512 7433 raft_consensus.cc:2781] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:50.243705 7430 leader_election.cc:288] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:50.243917 7434 raft_consensus.cc:2781] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:50.247262 7433 raft_consensus.cc:686] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Leader. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:11:50.250022 7430 ts_tablet_manager.cc:1398] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee: Time spent starting tablet: real 0.019s user 0.008s sys 0.009s
I20240503 02:11:50.248914 7433 consensus_queue.cc:237] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:50.250746 7430 tablet_bootstrap.cc:492] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: Bootstrap starting.
I20240503 02:11:50.252735 7430 tablet_bootstrap.cc:654] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:50.254160 7429 ts_tablet_manager.cc:1398] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70: Time spent starting tablet: real 0.023s user 0.014s sys 0.008s
I20240503 02:11:50.254372 7434 raft_consensus.cc:686] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Leader. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:50.254963 7434 consensus_queue.cc:237] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:50.258235 7430 tablet_bootstrap.cc:492] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: No bootstrap required, opened a new log
I20240503 02:11:50.258430 7430 ts_tablet_manager.cc:1367] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20240503 02:11:50.260574 7430 raft_consensus.cc:348] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:50.260856 7430 raft_consensus.cc:374] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:50.260983 7430 raft_consensus.cc:729] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Initialized, Role: FOLLOWER
I20240503 02:11:50.260950 6404 catalog_manager.cc:5484] T 8a3067767dbf425c8a2a662ec01aa935 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:50.261059 6403 catalog_manager.cc:5484] T d81a727ce55b4e629da976812205fb46 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: term changed from 0 to 1, leader changed from <none> to d305ba2a2a4640f7997613a7b8884dee (127.4.5.65). New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:50.261261 7430 consensus_queue.cc:260] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:50.261469 7430 raft_consensus.cc:388] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:50.261545 7430 raft_consensus.cc:482] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:50.261646 7430 raft_consensus.cc:3037] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:50.264353 7430 raft_consensus.cc:504] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:50.264688 7430 leader_election.cc:302] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:50.265038 7434 raft_consensus.cc:2781] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:50.265182 7434 raft_consensus.cc:686] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Leader. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:50.265393 7434 consensus_queue.cc:237] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:50.267791 7430 leader_election.cc:288] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:50.268543 7430 ts_tablet_manager.cc:1398] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: Time spent starting tablet: real 0.010s user 0.006s sys 0.000s
I20240503 02:11:50.269549 6404 catalog_manager.cc:5484] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: term changed from 0 to 1, leader changed from <none> to d305ba2a2a4640f7997613a7b8884dee (127.4.5.65). New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } health_report { overall_health: HEALTHY } } }
W20240503 02:11:50.680977 7445 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:50.681394 7445 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:50.689251 7445 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_0'
INSERT report
rows total: 2000
time total: 151.61 ms
time per row: 0.0758049 ms
I20240503 02:11:51.016541 6402 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57688:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:51.025938 6657 tablet_service.cc:1467] Processing CreateTablet for tablet 5c12714b4aab448e85708644e84e807c (DEFAULT_TABLE table=rebalance_test_table_1 [id=7dc9b08ad8464ad9b1733bc422522440]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:51.026162 6656 tablet_service.cc:1467] Processing CreateTablet for tablet 840968df19714728a1c194dc65fa1b82 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7dc9b08ad8464ad9b1733bc422522440]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:51.026268 6526 tablet_service.cc:1467] Processing CreateTablet for tablet 0b7371b6f21b4786b16678e0798be3a3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=7dc9b08ad8464ad9b1733bc422522440]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:51.026568 6657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c12714b4aab448e85708644e84e807c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:51.026798 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b7371b6f21b4786b16678e0798be3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:51.026901 6656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 840968df19714728a1c194dc65fa1b82. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:51.033609 7492 tablet_bootstrap.cc:492] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: Bootstrap starting.
I20240503 02:11:51.034543 7493 tablet_bootstrap.cc:492] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: Bootstrap starting.
I20240503 02:11:51.035429 7492 tablet_bootstrap.cc:654] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.036533 7493 tablet_bootstrap.cc:654] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.037941 7492 tablet_bootstrap.cc:492] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: No bootstrap required, opened a new log
I20240503 02:11:51.038082 7492 ts_tablet_manager.cc:1367] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:11:51.038699 7493 tablet_bootstrap.cc:492] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: No bootstrap required, opened a new log
I20240503 02:11:51.038843 7493 ts_tablet_manager.cc:1367] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.004s
I20240503 02:11:51.038975 7492 raft_consensus.cc:348] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.039145 7492 raft_consensus.cc:374] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.039242 7492 raft_consensus.cc:729] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.039494 7492 consensus_queue.cc:260] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.039541 7493 raft_consensus.cc:348] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.039660 7492 raft_consensus.cc:388] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:51.039731 7493 raft_consensus.cc:374] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.039746 7492 raft_consensus.cc:482] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:51.039832 7493 raft_consensus.cc:729] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.039851 7492 raft_consensus.cc:3037] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.040145 7493 consensus_queue.cc:260] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.040367 7493 raft_consensus.cc:388] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:51.040496 7493 raft_consensus.cc:482] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:51.040627 7493 raft_consensus.cc:3037] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.042264 7492 raft_consensus.cc:504] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.042513 7492 leader_election.cc:302] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:11:51.042706 7492 leader_election.cc:288] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:51.042848 7479 raft_consensus.cc:2781] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:51.042802 7493 raft_consensus.cc:504] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.043087 7493 leader_election.cc:302] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:51.043455 7493 leader_election.cc:288] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:51.043422 7479 raft_consensus.cc:686] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Leader. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:11:51.043576 7480 raft_consensus.cc:2781] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:51.043823 7479 consensus_queue.cc:237] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.044188 7493 ts_tablet_manager.cc:1398] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: Time spent starting tablet: real 0.005s user 0.000s sys 0.005s
I20240503 02:11:51.044637 7480 raft_consensus.cc:686] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Leader. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:51.045069 7480 consensus_queue.cc:237] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.050364 7492 ts_tablet_manager.cc:1398] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: Time spent starting tablet: real 0.012s user 0.004s sys 0.009s
I20240503 02:11:51.050805 7492 tablet_bootstrap.cc:492] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70: Bootstrap starting.
I20240503 02:11:51.051185 6402 catalog_manager.cc:5484] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:51.053877 7492 tablet_bootstrap.cc:654] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.056880 7492 tablet_bootstrap.cc:492] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70: No bootstrap required, opened a new log
I20240503 02:11:51.057075 7492 ts_tablet_manager.cc:1367] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:11:51.057936 7492 raft_consensus.cc:348] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.058162 7492 raft_consensus.cc:374] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.058254 7492 raft_consensus.cc:729] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.058570 7492 consensus_queue.cc:260] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.058746 7492 raft_consensus.cc:388] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:51.058861 7492 raft_consensus.cc:482] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:51.059002 7492 raft_consensus.cc:3037] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.061125 6402 catalog_manager.cc:5484] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: term changed from 0 to 1, leader changed from <none> to d305ba2a2a4640f7997613a7b8884dee (127.4.5.65). New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:51.062206 7492 raft_consensus.cc:504] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.063452 7492 leader_election.cc:302] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:11:51.063720 7492 leader_election.cc:288] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:51.063881 7497 raft_consensus.cc:2781] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:51.064370 7492 ts_tablet_manager.cc:1398] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20240503 02:11:51.064725 7497 raft_consensus.cc:686] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Leader. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:11:51.065074 7497 consensus_queue.cc:237] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.068009 6402 catalog_manager.cc:5484] T 840968df19714728a1c194dc65fa1b82 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } health_report { overall_health: HEALTHY } } }
W20240503 02:11:51.468765 7503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:51.469218 7503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:51.477581 7503 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_1'
INSERT report
rows total: 2000
time total: 145.614 ms
time per row: 0.072807 ms
I20240503 02:11:51.776600 6402 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57688:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:51.786113 6526 tablet_service.cc:1467] Processing CreateTablet for tablet 0d9322f2472b4ef6a8e808b5c91948d3 (DEFAULT_TABLE table=rebalance_test_table_2 [id=efe14e9c57a44f339179a831adbcd921]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:51.786280 6656 tablet_service.cc:1467] Processing CreateTablet for tablet 9bd51c9555b648d1baf4d1dcddc98fd6 (DEFAULT_TABLE table=rebalance_test_table_2 [id=efe14e9c57a44f339179a831adbcd921]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:51.786505 6525 tablet_service.cc:1467] Processing CreateTablet for tablet ff6cd0b970c5434bac564d6c1b2b20e1 (DEFAULT_TABLE table=rebalance_test_table_2 [id=efe14e9c57a44f339179a831adbcd921]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:51.786717 6656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd51c9555b648d1baf4d1dcddc98fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:51.786726 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9322f2472b4ef6a8e808b5c91948d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:51.786928 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff6cd0b970c5434bac564d6c1b2b20e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:51.793547 7551 tablet_bootstrap.cc:492] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: Bootstrap starting.
I20240503 02:11:51.794399 7552 tablet_bootstrap.cc:492] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: Bootstrap starting.
I20240503 02:11:51.795349 7551 tablet_bootstrap.cc:654] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.796329 7552 tablet_bootstrap.cc:654] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.797523 7551 tablet_bootstrap.cc:492] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: No bootstrap required, opened a new log
I20240503 02:11:51.797678 7551 ts_tablet_manager.cc:1367] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:11:51.798377 7552 tablet_bootstrap.cc:492] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: No bootstrap required, opened a new log
I20240503 02:11:51.798310 7551 raft_consensus.cc:348] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.798491 7551 raft_consensus.cc:374] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.798519 7552 ts_tablet_manager.cc:1367] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:51.798599 7551 raft_consensus.cc:729] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.798818 7551 consensus_queue.cc:260] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.799000 7551 raft_consensus.cc:388] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:51.799104 7551 raft_consensus.cc:482] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:51.799211 7551 raft_consensus.cc:3037] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.799265 7552 raft_consensus.cc:348] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.799439 7552 raft_consensus.cc:374] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.799533 7552 raft_consensus.cc:729] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.799795 7552 consensus_queue.cc:260] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.799961 7552 raft_consensus.cc:388] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:51.800050 7552 raft_consensus.cc:482] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:51.800185 7552 raft_consensus.cc:3037] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.801482 7551 raft_consensus.cc:504] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.801702 7551 leader_election.cc:302] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:51.801863 7551 leader_election.cc:288] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:51.801985 7538 raft_consensus.cc:2781] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:51.802500 7538 raft_consensus.cc:686] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Leader. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:51.802978 7538 consensus_queue.cc:237] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.803166 7552 raft_consensus.cc:504] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.803364 7552 leader_election.cc:302] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:11:51.803505 7552 leader_election.cc:288] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:51.803624 7539 raft_consensus.cc:2781] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:51.803975 7552 ts_tablet_manager.cc:1398] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:51.804744 7539 raft_consensus.cc:686] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Leader. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:11:51.805059 7539 consensus_queue.cc:237] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:51.807976 6402 catalog_manager.cc:5484] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:51.816700 7551 ts_tablet_manager.cc:1398] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: Time spent starting tablet: real 0.019s user 0.004s sys 0.015s
I20240503 02:11:51.817076 7551 tablet_bootstrap.cc:492] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: Bootstrap starting.
I20240503 02:11:51.818279 6402 catalog_manager.cc:5484] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: term changed from 0 to 1, leader changed from <none> to d305ba2a2a4640f7997613a7b8884dee (127.4.5.65). New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:51.818938 7551 tablet_bootstrap.cc:654] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:51.821190 7551 tablet_bootstrap.cc:492] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: No bootstrap required, opened a new log
I20240503 02:11:51.821384 7551 ts_tablet_manager.cc:1367] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:11:51.822158 7551 raft_consensus.cc:348] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.822356 7551 raft_consensus.cc:374] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:51.822438 7551 raft_consensus.cc:729] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Initialized, Role: FOLLOWER
I20240503 02:11:51.822715 7551 consensus_queue.cc:260] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.822889 7551 raft_consensus.cc:388] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:51.822991 7551 raft_consensus.cc:482] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:51.823086 7551 raft_consensus.cc:3037] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:51.825429 7551 raft_consensus.cc:504] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.825733 7551 leader_election.cc:302] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:51.826012 7551 leader_election.cc:288] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:51.826038 7556 raft_consensus.cc:2781] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:51.826224 7556 raft_consensus.cc:686] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Leader. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:51.826560 7556 consensus_queue.cc:237] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:51.826869 7551 ts_tablet_manager.cc:1398] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:51.829198 6402 catalog_manager.cc:5484] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: term changed from 0 to 1, leader changed from <none> to d305ba2a2a4640f7997613a7b8884dee (127.4.5.65). New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } health_report { overall_health: HEALTHY } } }
W20240503 02:11:52.216640 7562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:52.217046 7562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:52.225123 7562 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
INSERT report
rows total: 2000
time total: 132.363 ms
time per row: 0.0661817 ms
I20240503 02:11:52.514672 6403 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57688:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:52.524384 6656 tablet_service.cc:1467] Processing CreateTablet for tablet 38eaf65d847745e49865a6cff720cd7e (DEFAULT_TABLE table=rebalance_test_table_3 [id=107814137f694dceb7cfcaae16c5fd8f]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.524505 6657 tablet_service.cc:1467] Processing CreateTablet for tablet 6caaa6a65dae42c994a6c451a34a41d0 (DEFAULT_TABLE table=rebalance_test_table_3 [id=107814137f694dceb7cfcaae16c5fd8f]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.524993 6656 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eaf65d847745e49865a6cff720cd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.524886 6525 tablet_service.cc:1467] Processing CreateTablet for tablet 1c6cdade9a8241e4a0808deded2933f9 (DEFAULT_TABLE table=rebalance_test_table_3 [id=107814137f694dceb7cfcaae16c5fd8f]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:52.525244 6657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caaa6a65dae42c994a6c451a34a41d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.525343 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6cdade9a8241e4a0808deded2933f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:52.531930 7606 tablet_bootstrap.cc:492] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: Bootstrap starting.
I20240503 02:11:52.532352 7607 tablet_bootstrap.cc:492] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: Bootstrap starting.
I20240503 02:11:52.533788 7606 tablet_bootstrap.cc:654] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.534158 7607 tablet_bootstrap.cc:654] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.535660 7606 tablet_bootstrap.cc:492] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: No bootstrap required, opened a new log
I20240503 02:11:52.535825 7606 ts_tablet_manager.cc:1367] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:52.535892 7607 tablet_bootstrap.cc:492] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: No bootstrap required, opened a new log
I20240503 02:11:52.536038 7607 ts_tablet_manager.cc:1367] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:52.536554 7606 raft_consensus.cc:348] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:52.536720 7606 raft_consensus.cc:374] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.536715 7607 raft_consensus.cc:348] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:52.536823 7606 raft_consensus.cc:729] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.536895 7607 raft_consensus.cc:374] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.537034 7607 raft_consensus.cc:729] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.537075 7606 consensus_queue.cc:260] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:52.537290 7606 raft_consensus.cc:388] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:52.537263 7607 consensus_queue.cc:260] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:52.537390 7606 raft_consensus.cc:482] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:52.537477 7607 raft_consensus.cc:388] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:52.537497 7606 raft_consensus.cc:3037] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.537583 7607 raft_consensus.cc:482] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:52.537715 7607 raft_consensus.cc:3037] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.539609 7606 raft_consensus.cc:504] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:52.539638 7607 raft_consensus.cc:504] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:52.539865 7606 leader_election.cc:302] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:11:52.539870 7607 leader_election.cc:302] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:52.540030 7607 leader_election.cc:288] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:52.540042 7606 leader_election.cc:288] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:52.540132 7596 raft_consensus.cc:2781] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:52.540596 7593 raft_consensus.cc:2781] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:52.541107 7593 raft_consensus.cc:686] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Leader. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:11:52.541220 7607 ts_tablet_manager.cc:1398] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:11:52.541519 7593 consensus_queue.cc:237] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:52.545995 7596 raft_consensus.cc:686] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Leader. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:52.546437 7606 ts_tablet_manager.cc:1398] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: Time spent starting tablet: real 0.010s user 0.004s sys 0.007s
I20240503 02:11:52.546352 7596 consensus_queue.cc:237] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:52.546720 7606 tablet_bootstrap.cc:492] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: Bootstrap starting.
I20240503 02:11:52.546746 6403 catalog_manager.cc:5484] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:52.548368 7606 tablet_bootstrap.cc:654] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:52.550861 7606 tablet_bootstrap.cc:492] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: No bootstrap required, opened a new log
I20240503 02:11:52.551003 7606 ts_tablet_manager.cc:1367] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20240503 02:11:52.551640 7606 raft_consensus.cc:348] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:52.551816 7606 raft_consensus.cc:374] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:52.551894 7606 raft_consensus.cc:729] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Initialized, Role: FOLLOWER
I20240503 02:11:52.552129 7606 consensus_queue.cc:260] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:52.552263 7606 raft_consensus.cc:388] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:52.552364 7606 raft_consensus.cc:482] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:52.552479 7606 raft_consensus.cc:3037] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:52.554487 7606 raft_consensus.cc:504] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:52.554699 7606 leader_election.cc:302] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:11:52.554916 7606 leader_election.cc:288] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:52.555029 7611 raft_consensus.cc:2781] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:52.555243 7611 raft_consensus.cc:686] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Leader. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:11:52.555378 7606 ts_tablet_manager.cc:1398] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:11:52.555544 7611 consensus_queue.cc:237] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:52.560940 6403 catalog_manager.cc:5484] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:52.563454 6402 catalog_manager.cc:5484] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: term changed from 0 to 1, leader changed from <none> to d305ba2a2a4640f7997613a7b8884dee (127.4.5.65). New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } health_report { overall_health: HEALTHY } } }
W20240503 02:11:52.955647 7618 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:52.956120 7618 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:52.964385 7618 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_3'
INSERT report
rows total: 2000
time total: 137.718 ms
time per row: 0.0688591 ms
I20240503 02:11:53.257114 6400 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57688:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 1
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:11:53.267020 6525 tablet_service.cc:1467] Processing CreateTablet for tablet b630de5974f3462697dc12edee49e592 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0b61eafd89d498db460c250fdbaa217]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.267364 6657 tablet_service.cc:1467] Processing CreateTablet for tablet c041465d7d0f44f2af2beff9ed50ab96 (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0b61eafd89d498db460c250fdbaa217]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.267660 6525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b630de5974f3462697dc12edee49e592. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.267843 6657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c041465d7d0f44f2af2beff9ed50ab96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.267853 6526 tablet_service.cc:1467] Processing CreateTablet for tablet fe9f9a1a82914859bd08dd9f41de618a (DEFAULT_TABLE table=rebalance_test_table_4 [id=c0b61eafd89d498db460c250fdbaa217]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:11:53.268193 6526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9f9a1a82914859bd08dd9f41de618a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:53.281127 7665 tablet_bootstrap.cc:492] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: Bootstrap starting.
I20240503 02:11:53.281754 7664 tablet_bootstrap.cc:492] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: Bootstrap starting.
I20240503 02:11:53.283593 7665 tablet_bootstrap.cc:654] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.283803 7664 tablet_bootstrap.cc:654] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.286114 7665 tablet_bootstrap.cc:492] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: No bootstrap required, opened a new log
I20240503 02:11:53.286291 7665 ts_tablet_manager.cc:1367] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:11:53.286374 7664 tablet_bootstrap.cc:492] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: No bootstrap required, opened a new log
I20240503 02:11:53.286535 7664 ts_tablet_manager.cc:1367] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:11:53.287071 7665 raft_consensus.cc:348] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:53.287262 7665 raft_consensus.cc:374] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.287361 7665 raft_consensus.cc:729] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.287612 7665 consensus_queue.cc:260] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:53.287729 7664 raft_consensus.cc:348] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:53.287834 7665 raft_consensus.cc:388] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:53.287920 7664 raft_consensus.cc:374] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.287916 7665 raft_consensus.cc:482] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:53.288003 7664 raft_consensus.cc:729] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.288022 7665 raft_consensus.cc:3037] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.288266 7664 consensus_queue.cc:260] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:53.288475 7664 raft_consensus.cc:388] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:53.288550 7664 raft_consensus.cc:482] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:53.288648 7664 raft_consensus.cc:3037] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.290613 7665 raft_consensus.cc:504] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:53.290939 7665 leader_election.cc:302] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:11:53.291199 7665 leader_election.cc:288] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:53.291333 7651 raft_consensus.cc:2781] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:53.291801 7665 ts_tablet_manager.cc:1398] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: Time spent starting tablet: real 0.005s user 0.003s sys 0.002s
I20240503 02:11:53.292238 7651 raft_consensus.cc:686] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Leader. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:11:53.292616 7651 consensus_queue.cc:237] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } }
I20240503 02:11:53.292707 7664 raft_consensus.cc:504] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:53.292997 7664 leader_election.cc:302] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:53.293188 7664 leader_election.cc:288] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:53.293412 7653 raft_consensus.cc:2781] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:53.305428 7653 raft_consensus.cc:686] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Leader. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:53.306007 7653 consensus_queue.cc:237] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:53.307152 7664 ts_tablet_manager.cc:1398] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: Time spent starting tablet: real 0.020s user 0.007s sys 0.011s
I20240503 02:11:53.307528 7664 tablet_bootstrap.cc:492] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee: Bootstrap starting.
I20240503 02:11:53.309651 7664 tablet_bootstrap.cc:654] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee: Neither blocks nor log segments found. Creating new log.
I20240503 02:11:53.310108 6400 catalog_manager.cc:5484] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:53.317068 6397 catalog_manager.cc:5484] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee reported cstate change: term changed from 0 to 1, leader changed from <none> to d305ba2a2a4640f7997613a7b8884dee (127.4.5.65). New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:53.322943 7664 tablet_bootstrap.cc:492] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee: No bootstrap required, opened a new log
I20240503 02:11:53.323149 7664 ts_tablet_manager.cc:1367] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.009s
I20240503 02:11:53.324100 7664 raft_consensus.cc:348] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:53.324301 7664 raft_consensus.cc:374] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:11:53.324363 7664 raft_consensus.cc:729] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Initialized, Role: FOLLOWER
I20240503 02:11:53.324659 7664 consensus_queue.cc:260] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:53.324828 7664 raft_consensus.cc:388] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:53.324940 7664 raft_consensus.cc:482] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:53.325039 7664 raft_consensus.cc:3037] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:11:53.327590 7664 raft_consensus.cc:504] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:53.327826 7664 leader_election.cc:302] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:53.328006 7664 leader_election.cc:288] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:11:53.328109 7653 raft_consensus.cc:2781] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:11:53.328349 7653 raft_consensus.cc:686] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Leader. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:53.328491 7664 ts_tablet_manager.cc:1398] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee: Time spent starting tablet: real 0.005s user 0.006s sys 0.000s
I20240503 02:11:53.328672 7653 consensus_queue.cc:237] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } }
I20240503 02:11:53.331426 6397 catalog_manager.cc:5484] T b630de5974f3462697dc12edee49e592 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: term changed from 0 to 1, leader changed from <none> to d305ba2a2a4640f7997613a7b8884dee (127.4.5.65). New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } health_report { overall_health: HEALTHY } } }
W20240503 02:11:53.716218 7677 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:53.716647 7677 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:53.724414 7677 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
INSERT report
rows total: 2000
time total: 155.193 ms
time per row: 0.0775965 ms
I20240503 02:11:54.031013 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.67:41365
--local_ip_for_outbound_sockets=127.4.5.67
--tserver_master_addrs=127.4.5.126:42813
--webserver_port=38591
--webserver_interface=127.4.5.67
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:11:54.406265 7720 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:54.406622 7720 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:54.406737 7720 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:54.415634 7720 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:54.415774 7720 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:54.415932 7720 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.67
I20240503 02:11:54.425724 7720 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.67:41365
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.5.67
--webserver_port=38591
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.67
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:54.427332 7720 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:54.429601 7720 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:54.443532 7726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:54.444146 7725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:54.444854 7728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:54.445587 7720 server_base.cc:1034] running on GCE node
I20240503 02:11:54.446343 7720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:54.447439 7720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:54.448647 7720 hybrid_clock.cc:648] HybridClock initialized: now 1714702314448595 us; error 51 us; skew 500 ppm
I20240503 02:11:54.449000 7720 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:54.451611 7720 webserver.cc:460] Webserver started at http://127.4.5.67:38591/ using document root <none> and password file <none>
I20240503 02:11:54.452498 7720 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:54.452644 7720 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:54.458379 7720 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:11:54.461310 7734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:54.462843 7720 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:54.462989 7720 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "5ad3a6abdbc846c1a6ed42269a100dd3"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-zv3f"
I20240503 02:11:54.463891 7720 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:54.489606 7720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:54.490656 7720 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:54.490958 7720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:54.492007 7720 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:54.493772 7720 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:54.493846 7720 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:54.493947 7720 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:54.493999 7720 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:54.528122 7720 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.67:41365
I20240503 02:11:54.528141 7849 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.67:41365 every 8 connection(s)
I20240503 02:11:54.529817 7720 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:11:54.533556 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 7720
I20240503 02:11:54.534196 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.68:42071
--local_ip_for_outbound_sockets=127.4.5.68
--tserver_master_addrs=127.4.5.126:42813
--webserver_port=33247
--webserver_interface=127.4.5.68
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:54.544806 7850 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:54.545243 7850 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:54.546515 7850 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:54.548843 6397 ts_manager.cc:194] Re-registered known tserver with Master: 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67:41365)
I20240503 02:11:54.550462 6397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.67:55629
W20240503 02:11:54.906507 7854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:54.906870 7854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:54.907032 7854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:54.915278 7854 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:54.915441 7854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:54.915619 7854 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.68
I20240503 02:11:54.925642 7854 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.68:42071
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.4.5.68
--webserver_port=33247
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.68
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:54.927439 7854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:54.929663 7854 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:54.943500 7861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:54.944188 7863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:54.944689 7860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:54.945397 7854 server_base.cc:1034] running on GCE node
I20240503 02:11:54.946090 7854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:54.947229 7854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:54.948428 7854 hybrid_clock.cc:648] HybridClock initialized: now 1714702314948357 us; error 62 us; skew 500 ppm
I20240503 02:11:54.948863 7854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:54.951329 7854 webserver.cc:460] Webserver started at http://127.4.5.68:33247/ using document root <none> and password file <none>
I20240503 02:11:54.952162 7854 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:54.952299 7854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:54.957880 7854 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20240503 02:11:54.960940 7870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:54.962335 7854 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:54.962507 7854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "6f857824c6df4cb3a0eb69a4b66810f9"
format_stamp: "Formatted at 2024-05-03 02:11:43 on dist-test-slave-zv3f"
I20240503 02:11:54.963603 7854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:54.992620 7854 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:54.993594 7854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:54.993883 7854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:54.994897 7854 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:54.997082 7877 ts_tablet_manager.cc:522] Loading tablet metadata (0/1 complete)
I20240503 02:11:55.003275 7854 ts_tablet_manager.cc:559] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20240503 02:11:55.003396 7854 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.007s user 0.001s sys 0.000s
I20240503 02:11:55.003556 7854 ts_tablet_manager.cc:574] Registering tablets (0/1 complete)
I20240503 02:11:55.006731 7854 ts_tablet_manager.cc:590] Registered 1 tablets
I20240503 02:11:55.006857 7854 ts_tablet_manager.cc:569] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20240503 02:11:55.007215 7877 tablet_bootstrap.cc:492] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Bootstrap starting.
I20240503 02:11:55.028757 7877 log.cc:824] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:55.036422 7877 tablet_bootstrap.cc:492] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=1 overwritten=0 applied=1 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:55.037604 7877 tablet_bootstrap.cc:492] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Bootstrap complete.
I20240503 02:11:55.039214 7877 ts_tablet_manager.cc:1367] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Time spent bootstrapping tablet: real 0.032s user 0.013s sys 0.016s
I20240503 02:11:55.049921 7877 raft_consensus.cc:348] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } }
I20240503 02:11:55.051317 7877 raft_consensus.cc:729] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f857824c6df4cb3a0eb69a4b66810f9, State: Initialized, Role: FOLLOWER
I20240503 02:11:55.052121 7877 consensus_queue.cc:260] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } }
I20240503 02:11:55.052378 7877 raft_consensus.cc:388] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:11:55.052551 7877 raft_consensus.cc:482] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:11:55.052779 7877 raft_consensus.cc:3037] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:55.057497 7854 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.68:42071
I20240503 02:11:55.057528 7986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.68:42071 every 8 connection(s)
I20240503 02:11:55.058066 7877 raft_consensus.cc:504] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } }
I20240503 02:11:55.058610 7877 leader_election.cc:302] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f857824c6df4cb3a0eb69a4b66810f9; no voters:
I20240503 02:11:55.059137 7854 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:11:55.059221 7877 leader_election.cc:288] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [CANDIDATE]: Term 2 election: Requested vote from peers
I20240503 02:11:55.059379 7987 raft_consensus.cc:2781] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:55.061228 7877 ts_tablet_manager.cc:1398] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9: Time spent starting tablet: real 0.022s user 0.021s sys 0.000s
I20240503 02:11:55.061628 7987 raft_consensus.cc:686] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [term 2 LEADER]: Becoming Leader. State: Replica: 6f857824c6df4cb3a0eb69a4b66810f9, State: Running, Role: LEADER
I20240503 02:11:55.062319 7987 consensus_queue.cc:237] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } }
I20240503 02:11:55.068986 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 7854
I20240503 02:11:55.069841 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.69:35867
--local_ip_for_outbound_sockets=127.4.5.69
--tserver_master_addrs=127.4.5.126:42813
--webserver_port=37647
--webserver_interface=127.4.5.69
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:55.078022 7990 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:55.078330 7990 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:55.079298 7990 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:55.083423 6397 ts_manager.cc:194] Re-registered known tserver with Master: 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68:42071)
I20240503 02:11:55.083878 6397 catalog_manager.cc:5484] T 83124ba50be244e3bcc43c41e9b644ec P 6f857824c6df4cb3a0eb69a4b66810f9 reported cstate change: term changed from 1 to 2. New cstate: current_term: 2 leader_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } health_report { overall_health: HEALTHY } } }
I20240503 02:11:55.089644 6397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.68:48999
I20240503 02:11:55.092734 7990 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
W20240503 02:11:55.441287 7996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:55.441624 7996 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:55.441799 7996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:55.450296 7996 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:55.450438 7996 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:55.450606 7996 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.69
I20240503 02:11:55.460156 7996 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.69:35867
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.4.5.69
--webserver_port=37647
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.69
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:55.461906 7996 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:55.463858 7996 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:55.477722 8006 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:55.478226 8005 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:55.478765 8008 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:55.479682 7996 server_base.cc:1034] running on GCE node
I20240503 02:11:55.480373 7996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:55.481513 7996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:55.482757 7996 hybrid_clock.cc:648] HybridClock initialized: now 1714702315482701 us; error 60 us; skew 500 ppm
I20240503 02:11:55.483111 7996 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:55.485714 7996 webserver.cc:460] Webserver started at http://127.4.5.69:37647/ using document root <none> and password file <none>
I20240503 02:11:55.486565 7996 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:55.486706 7996 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:55.492432 7996 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20240503 02:11:55.495524 8014 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:55.497004 7996 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:11:55.497175 7996 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e3fafd8e3e974740b2525f2e89d7cca5"
format_stamp: "Formatted at 2024-05-03 02:11:44 on dist-test-slave-zv3f"
I20240503 02:11:55.498077 7996 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:55.515165 7996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:55.516068 7996 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:55.516366 7996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:55.517376 7996 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:55.518983 7996 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:55.519065 7996 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:55.519189 7996 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:55.519244 7996 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:55.553988 7996 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.69:35867
I20240503 02:11:55.554008 8128 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.69:35867 every 8 connection(s)
I20240503 02:11:55.554245 7850 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:55.556298 7996 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:11:55.563365 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 7996
I20240503 02:11:55.564054 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.70:40279
--local_ip_for_outbound_sockets=127.4.5.70
--tserver_master_addrs=127.4.5.126:42813
--webserver_port=35955
--webserver_interface=127.4.5.70
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:55.573762 8129 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:55.574119 8129 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:55.575145 8129 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:55.576901 6397 ts_manager.cc:194] Re-registered known tserver with Master: e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69:35867)
I20240503 02:11:55.577862 6397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.69:46445
W20240503 02:11:55.925523 8133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:55.925863 8133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:55.925988 8133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:55.933534 8133 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:55.933672 8133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:55.933804 8133 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.70
I20240503 02:11:55.943087 8133 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.70:40279
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.4.5.70
--webserver_port=35955
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.70
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:55.944741 8133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:55.946770 8133 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:55.959832 8138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:55.960372 8139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:55.961022 8141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:55.961799 8133 server_base.cc:1034] running on GCE node
I20240503 02:11:55.962441 8133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:55.963459 8133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:55.964663 8133 hybrid_clock.cc:648] HybridClock initialized: now 1714702315964618 us; error 34 us; skew 500 ppm
I20240503 02:11:55.965011 8133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:55.972983 8133 webserver.cc:460] Webserver started at http://127.4.5.70:35955/ using document root <none> and password file <none>
I20240503 02:11:55.973850 8133 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:55.973994 8133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:55.979873 8133 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20240503 02:11:55.982932 8147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:55.984351 8133 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:11:55.984536 8133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "329369f9124942438b3176e3b0bacf98"
format_stamp: "Formatted at 2024-05-03 02:11:44 on dist-test-slave-zv3f"
I20240503 02:11:55.985441 8133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:56.032011 8133 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:56.033247 8133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:56.033601 8133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:56.034756 8133 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:56.036967 8133 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:56.037068 8133 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:56.037212 8133 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:56.037288 8133 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:56.083937 8133 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.70:40279
I20240503 02:11:56.084020 8262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.70:40279 every 8 connection(s)
I20240503 02:11:56.086032 8133 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:11:56.087653 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 8133
I20240503 02:11:56.088279 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.71:37017
--local_ip_for_outbound_sockets=127.4.5.71
--tserver_master_addrs=127.4.5.126:42813
--webserver_port=33157
--webserver_interface=127.4.5.71
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:11:56.120599 8264 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:56.120975 8264 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:56.121984 8264 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:56.123585 6397 ts_manager.cc:194] Re-registered known tserver with Master: 329369f9124942438b3176e3b0bacf98 (127.4.5.70:40279)
I20240503 02:11:56.124506 6397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.70:41377
W20240503 02:11:56.472985 8268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:11:56.473369 8268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:11:56.473528 8268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:11:56.481825 8268 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:11:56.482074 8268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:11:56.482260 8268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.71
I20240503 02:11:56.493656 8268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.71:37017
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.4.5.71
--webserver_port=33157
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.71
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:11:56.495385 8268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:11:56.497550 8268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:11:56.512966 8273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:56.513506 8274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:11:56.514160 8276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:11:56.515034 8268 server_base.cc:1034] running on GCE node
I20240503 02:11:56.515812 8268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:11:56.517128 8268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:11:56.518333 8268 hybrid_clock.cc:648] HybridClock initialized: now 1714702316518257 us; error 64 us; skew 500 ppm
I20240503 02:11:56.518697 8268 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:11:56.521363 8268 webserver.cc:460] Webserver started at http://127.4.5.71:33157/ using document root <none> and password file <none>
I20240503 02:11:56.522260 8268 fs_manager.cc:359] Metadata directory not provided
I20240503 02:11:56.522393 8268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:11:56.528801 8268 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20240503 02:11:56.532573 8282 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:56.534139 8268 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:11:56.534314 8268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "4bf9c9a4cde949dbabf5144188edbf5b"
format_stamp: "Formatted at 2024-05-03 02:11:45 on dist-test-slave-zv3f"
I20240503 02:11:56.535285 8268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:11:56.566005 8268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:11:56.567035 8268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:11:56.567370 8268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:11:56.568581 8268 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:11:56.570475 8268 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:11:56.570564 8268 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:56.570703 8268 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:11:56.570829 8268 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:11:56.580998 8129 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:56.613018 8268 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.71:37017
I20240503 02:11:56.613041 8395 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.71:37017 every 8 connection(s)
I20240503 02:11:56.614753 8268 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:11:56.618716 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 8268
I20240503 02:11:56.632242 8396 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:11:56.632755 8396 heartbeater.cc:464] Registering TS with master...
I20240503 02:11:56.634078 8396 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:11:56.636492 6397 ts_manager.cc:194] Re-registered known tserver with Master: 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71:37017)
I20240503 02:11:56.638052 6397 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.71:39889
I20240503 02:11:57.127418 8264 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:57.277774 8197 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.279227 8331 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.307628 7783 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.309481 6657 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:57.345093 7920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.376181 6526 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:57.397645 8064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.636673 8197 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.642905 8396 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:11:57.651031 8331 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.660753 6657 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:11:57.683475 7920 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:11:57.686882 7783 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.726220 6526 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:11:57.738734 8064 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:11:57.823004 6546 consensus_queue.cc:237] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } }
I20240503 02:11:57.826763 8491 raft_consensus.cc:2932] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } } }
I20240503 02:11:57.836926 6400 catalog_manager.cc:5484] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index -1 to 4, NON_VOTER 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:57.843015 6480 consensus_peers.cc:485] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee -> Peer 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67:41365): Couldn't send request to peer 5ad3a6abdbc846c1a6ed42269a100dd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fe9f9a1a82914859bd08dd9f41de618a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:57.859077 6677 consensus_queue.cc:237] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } }
I20240503 02:11:57.863232 8505 raft_consensus.cc:2932] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } } }
I20240503 02:11:57.871016 6400 catalog_manager.cc:5484] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index -1 to 4, NON_VOTER e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:57.875905 6613 consensus_peers.cc:485] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 -> Peer e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69:35867): Couldn't send request to peer e3fafd8e3e974740b2525f2e89d7cca5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 38eaf65d847745e49865a6cff720cd7e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:57.885257 6546 consensus_queue.cc:237] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } }
I20240503 02:11:57.887707 8490 raft_consensus.cc:2932] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } } }
I20240503 02:11:57.893617 6400 catalog_manager.cc:5484] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index -1 to 4, NON_VOTER 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:57.896730 6479 consensus_peers.cc:485] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee -> Peer 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71:37017): Couldn't send request to peer 4bf9c9a4cde949dbabf5144188edbf5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ff6cd0b970c5434bac564d6c1b2b20e1. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:57.909266 6677 consensus_queue.cc:237] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } }
I20240503 02:11:57.911826 8505 raft_consensus.cc:2932] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 329369f9124942438b3176e3b0bacf98 (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } } }
I20240503 02:11:57.917906 6400 catalog_manager.cc:5484] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index -1 to 4, NON_VOTER 329369f9124942438b3176e3b0bacf98 (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:57.921496 6610 consensus_peers.cc:485] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 -> Peer 329369f9124942438b3176e3b0bacf98 (127.4.5.70:40279): Couldn't send request to peer 329369f9124942438b3176e3b0bacf98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c12714b4aab448e85708644e84e807c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:57.931547 6546 consensus_queue.cc:237] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: true } }
I20240503 02:11:57.933912 8490 raft_consensus.cc:2932] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: true } } }
I20240503 02:11:57.940697 6400 catalog_manager.cc:5484] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index -1 to 4, NON_VOTER 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:57.942519 6482 consensus_peers.cc:485] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee -> Peer 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68:42071): Couldn't send request to peer 6f857824c6df4cb3a0eb69a4b66810f9. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 587895f39d8b47858e704e8ccb274b89. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:57.954023 6546 consensus_queue.cc:237] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } }
I20240503 02:11:57.956167 8490 raft_consensus.cc:2932] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } } }
W20240503 02:11:57.956672 6480 consensus_peers.cc:485] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee -> Peer 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67:41365): Couldn't send request to peer 5ad3a6abdbc846c1a6ed42269a100dd3. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0b7371b6f21b4786b16678e0798be3a3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:57.961616 6400 catalog_manager.cc:5484] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index -1 to 4, NON_VOTER 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:57.976384 6677 consensus_queue.cc:237] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } }
I20240503 02:11:57.978308 8505 raft_consensus.cc:2932] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } } }
W20240503 02:11:57.978973 6613 consensus_peers.cc:485] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 -> Peer e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69:35867): Couldn't send request to peer e3fafd8e3e974740b2525f2e89d7cca5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9bd51c9555b648d1baf4d1dcddc98fd6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:57.983744 6401 catalog_manager.cc:5484] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index -1 to 4, NON_VOTER e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:11:57.997467 6677 consensus_queue.cc:237] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } }
I20240503 02:11:57.999763 8539 raft_consensus.cc:2932] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } } }
I20240503 02:11:58.006601 6401 catalog_manager.cc:5484] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index -1 to 4, NON_VOTER 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:58.008049 6610 consensus_peers.cc:485] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 -> Peer 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71:37017): Couldn't send request to peer 4bf9c9a4cde949dbabf5144188edbf5b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c041465d7d0f44f2af2beff9ed50ab96. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:58.020220 6546 consensus_queue.cc:237] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } }
I20240503 02:11:58.022686 8490 raft_consensus.cc:2932] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 329369f9124942438b3176e3b0bacf98 (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } } }
I20240503 02:11:58.030004 6401 catalog_manager.cc:5484] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index -1 to 4, NON_VOTER 329369f9124942438b3176e3b0bacf98 (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:11:58.031057 6479 consensus_peers.cc:485] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee -> Peer 329369f9124942438b3176e3b0bacf98 (127.4.5.70:40279): Couldn't send request to peer 329369f9124942438b3176e3b0bacf98. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1c6cdade9a8241e4a0808deded2933f9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:11:58.328073 8600 ts_tablet_manager.cc:902] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: Initiating tablet copy from peer d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:11:58.330793 8600 tablet_copy_client.cc:308] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:39365
I20240503 02:11:58.340467 6566 tablet_copy_service.cc:140] P d305ba2a2a4640f7997613a7b8884dee: Received BeginTabletCopySession request for tablet 587895f39d8b47858e704e8ccb274b89 from peer 6f857824c6df4cb3a0eb69a4b66810f9 ({username='slave'} at 127.4.5.68:57827)
I20240503 02:11:58.340752 8607 ts_tablet_manager.cc:902] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: Initiating tablet copy from peer 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:11:58.340906 6566 tablet_copy_service.cc:161] P d305ba2a2a4640f7997613a7b8884dee: Beginning new tablet copy session on tablet 587895f39d8b47858e704e8ccb274b89 from peer 6f857824c6df4cb3a0eb69a4b66810f9 at {username='slave'} at 127.4.5.68:57827: session id = 6f857824c6df4cb3a0eb69a4b66810f9-587895f39d8b47858e704e8ccb274b89
I20240503 02:11:58.347764 6566 tablet_copy_source_session.cc:205] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.351068 8609 ts_tablet_manager.cc:902] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: Initiating tablet copy from peer d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:11:58.353251 8600 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587895f39d8b47858e704e8ccb274b89. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.367530 8600 tablet_copy_client.cc:783] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.369071 8600 tablet_copy_client.cc:647] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.373293 8600 tablet_copy_client.cc:523] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.379165 8600 tablet_bootstrap.cc:492] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: Bootstrap starting.
I20240503 02:11:58.386086 8609 tablet_copy_client.cc:308] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:39365
I20240503 02:11:58.409065 8607 tablet_copy_client.cc:308] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:39573
I20240503 02:11:58.412220 6566 tablet_copy_service.cc:140] P d305ba2a2a4640f7997613a7b8884dee: Received BeginTabletCopySession request for tablet ff6cd0b970c5434bac564d6c1b2b20e1 from peer 4bf9c9a4cde949dbabf5144188edbf5b ({username='slave'} at 127.4.5.71:37275)
I20240503 02:11:58.412539 6566 tablet_copy_service.cc:161] P d305ba2a2a4640f7997613a7b8884dee: Beginning new tablet copy session on tablet ff6cd0b970c5434bac564d6c1b2b20e1 from peer 4bf9c9a4cde949dbabf5144188edbf5b at {username='slave'} at 127.4.5.71:37275: session id = 4bf9c9a4cde949dbabf5144188edbf5b-ff6cd0b970c5434bac564d6c1b2b20e1
I20240503 02:11:58.417133 6566 tablet_copy_source_session.cc:205] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.424616 8609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ff6cd0b970c5434bac564d6c1b2b20e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.430786 8630 ts_tablet_manager.cc:902] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: Initiating tablet copy from peer 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:11:58.449949 8609 tablet_copy_client.cc:783] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.450623 8609 tablet_copy_client.cc:647] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.457538 8633 ts_tablet_manager.cc:902] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: Initiating tablet copy from peer d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:11:58.457790 6697 tablet_copy_service.cc:140] P 4e21bb5794994a489ebd09ee30b6df70: Received BeginTabletCopySession request for tablet 5c12714b4aab448e85708644e84e807c from peer 329369f9124942438b3176e3b0bacf98 ({username='slave'} at 127.4.5.70:37783)
I20240503 02:11:58.458228 6697 tablet_copy_service.cc:161] P 4e21bb5794994a489ebd09ee30b6df70: Beginning new tablet copy session on tablet 5c12714b4aab448e85708644e84e807c from peer 329369f9124942438b3176e3b0bacf98 at {username='slave'} at 127.4.5.70:37783: session id = 329369f9124942438b3176e3b0bacf98-5c12714b4aab448e85708644e84e807c
I20240503 02:11:58.464797 6697 tablet_copy_source_session.cc:205] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.469309 8609 tablet_copy_client.cc:523] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.472661 8600 tablet_bootstrap.cc:492] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.473390 8600 tablet_bootstrap.cc:492] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: Bootstrap complete.
I20240503 02:11:58.473888 8600 ts_tablet_manager.cc:1367] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: Time spent bootstrapping tablet: real 0.095s user 0.072s sys 0.017s
I20240503 02:11:58.474031 8607 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c12714b4aab448e85708644e84e807c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.475497 8600 raft_consensus.cc:348] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: true } }
I20240503 02:11:58.476054 8600 raft_consensus.cc:729] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6f857824c6df4cb3a0eb69a4b66810f9, State: Initialized, Role: LEARNER
I20240503 02:11:58.476492 8600 consensus_queue.cc:260] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: true } }
I20240503 02:11:58.477341 8609 tablet_bootstrap.cc:492] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: Bootstrap starting.
I20240503 02:11:58.484681 8600 ts_tablet_manager.cc:1398] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20240503 02:11:58.489542 8630 tablet_copy_client.cc:308] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:39573
I20240503 02:11:58.492349 6566 tablet_copy_service.cc:342] P d305ba2a2a4640f7997613a7b8884dee: Request end of tablet copy session 6f857824c6df4cb3a0eb69a4b66810f9-587895f39d8b47858e704e8ccb274b89 received from {username='slave'} at 127.4.5.68:57827
I20240503 02:11:58.492681 6566 tablet_copy_service.cc:433] P d305ba2a2a4640f7997613a7b8884dee: ending tablet copy session 6f857824c6df4cb3a0eb69a4b66810f9-587895f39d8b47858e704e8ccb274b89 on tablet 587895f39d8b47858e704e8ccb274b89 with peer 6f857824c6df4cb3a0eb69a4b66810f9
I20240503 02:11:58.503160 8646 ts_tablet_manager.cc:902] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: Initiating tablet copy from peer 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:11:58.505882 6697 tablet_copy_service.cc:140] P 4e21bb5794994a489ebd09ee30b6df70: Received BeginTabletCopySession request for tablet 38eaf65d847745e49865a6cff720cd7e from peer e3fafd8e3e974740b2525f2e89d7cca5 ({username='slave'} at 127.4.5.69:53415)
I20240503 02:11:58.506184 6697 tablet_copy_service.cc:161] P 4e21bb5794994a489ebd09ee30b6df70: Beginning new tablet copy session on tablet 38eaf65d847745e49865a6cff720cd7e from peer e3fafd8e3e974740b2525f2e89d7cca5 at {username='slave'} at 127.4.5.69:53415: session id = e3fafd8e3e974740b2525f2e89d7cca5-38eaf65d847745e49865a6cff720cd7e
I20240503 02:11:58.514978 6697 tablet_copy_source_session.cc:205] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.517518 8633 tablet_copy_client.cc:308] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:39365
I20240503 02:11:58.522585 8607 tablet_copy_client.cc:783] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.523154 8607 tablet_copy_client.cc:647] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.524199 8609 log.cc:824] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:58.524730 8630 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eaf65d847745e49865a6cff720cd7e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.532727 8607 tablet_copy_client.cc:523] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.534938 8646 tablet_copy_client.cc:308] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:39573
I20240503 02:11:58.540144 6566 tablet_copy_service.cc:140] P d305ba2a2a4640f7997613a7b8884dee: Received BeginTabletCopySession request for tablet fe9f9a1a82914859bd08dd9f41de618a from peer 5ad3a6abdbc846c1a6ed42269a100dd3 ({username='slave'} at 127.4.5.67:50105)
I20240503 02:11:58.540395 6566 tablet_copy_service.cc:161] P d305ba2a2a4640f7997613a7b8884dee: Beginning new tablet copy session on tablet fe9f9a1a82914859bd08dd9f41de618a from peer 5ad3a6abdbc846c1a6ed42269a100dd3 at {username='slave'} at 127.4.5.67:50105: session id = 5ad3a6abdbc846c1a6ed42269a100dd3-fe9f9a1a82914859bd08dd9f41de618a
I20240503 02:11:58.544623 6697 tablet_copy_service.cc:140] P 4e21bb5794994a489ebd09ee30b6df70: Received BeginTabletCopySession request for tablet c041465d7d0f44f2af2beff9ed50ab96 from peer 4bf9c9a4cde949dbabf5144188edbf5b ({username='slave'} at 127.4.5.71:56097)
I20240503 02:11:58.544853 6697 tablet_copy_service.cc:161] P 4e21bb5794994a489ebd09ee30b6df70: Beginning new tablet copy session on tablet c041465d7d0f44f2af2beff9ed50ab96 from peer 4bf9c9a4cde949dbabf5144188edbf5b at {username='slave'} at 127.4.5.71:56097: session id = 4bf9c9a4cde949dbabf5144188edbf5b-c041465d7d0f44f2af2beff9ed50ab96
I20240503 02:11:58.544966 8630 tablet_copy_client.cc:783] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.545064 6566 tablet_copy_source_session.cc:205] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.545396 8630 tablet_copy_client.cc:647] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.546422 8607 tablet_bootstrap.cc:492] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: Bootstrap starting.
I20240503 02:11:58.548584 6697 tablet_copy_source_session.cc:205] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.550371 8646 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c041465d7d0f44f2af2beff9ed50ab96. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.553054 8633 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fe9f9a1a82914859bd08dd9f41de618a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.553473 8630 tablet_copy_client.cc:523] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.559134 8630 tablet_bootstrap.cc:492] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: Bootstrap starting.
I20240503 02:11:58.594434 8630 log.cc:824] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:58.597357 8633 tablet_copy_client.cc:783] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.598225 8633 tablet_copy_client.cc:647] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.603118 8633 tablet_copy_client.cc:523] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.608220 8633 tablet_bootstrap.cc:492] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: Bootstrap starting.
I20240503 02:11:58.608273 8646 tablet_copy_client.cc:783] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.619575 8661 ts_tablet_manager.cc:902] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: Initiating tablet copy from peer d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:11:58.621680 8663 ts_tablet_manager.cc:902] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: Initiating tablet copy from peer 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:11:58.629326 8646 tablet_copy_client.cc:647] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.623776 8660 ts_tablet_manager.cc:902] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: Initiating tablet copy from peer d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:11:58.631477 8607 log.cc:824] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:58.632627 8646 tablet_copy_client.cc:523] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.641902 8646 tablet_bootstrap.cc:492] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: Bootstrap starting.
I20240503 02:11:58.666798 8663 tablet_copy_client.cc:308] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:39573
I20240503 02:11:58.667928 6697 tablet_copy_service.cc:140] P 4e21bb5794994a489ebd09ee30b6df70: Received BeginTabletCopySession request for tablet 9bd51c9555b648d1baf4d1dcddc98fd6 from peer e3fafd8e3e974740b2525f2e89d7cca5 ({username='slave'} at 127.4.5.69:53415)
I20240503 02:11:58.668210 6697 tablet_copy_service.cc:161] P 4e21bb5794994a489ebd09ee30b6df70: Beginning new tablet copy session on tablet 9bd51c9555b648d1baf4d1dcddc98fd6 from peer e3fafd8e3e974740b2525f2e89d7cca5 at {username='slave'} at 127.4.5.69:53415: session id = e3fafd8e3e974740b2525f2e89d7cca5-9bd51c9555b648d1baf4d1dcddc98fd6
I20240503 02:11:58.671540 6697 tablet_copy_source_session.cc:205] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.673435 8663 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bd51c9555b648d1baf4d1dcddc98fd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.676018 8660 tablet_copy_client.cc:308] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:39365
I20240503 02:11:58.677240 6566 tablet_copy_service.cc:140] P d305ba2a2a4640f7997613a7b8884dee: Received BeginTabletCopySession request for tablet 0b7371b6f21b4786b16678e0798be3a3 from peer 5ad3a6abdbc846c1a6ed42269a100dd3 ({username='slave'} at 127.4.5.67:50105)
I20240503 02:11:58.677529 6566 tablet_copy_service.cc:161] P d305ba2a2a4640f7997613a7b8884dee: Beginning new tablet copy session on tablet 0b7371b6f21b4786b16678e0798be3a3 from peer 5ad3a6abdbc846c1a6ed42269a100dd3 at {username='slave'} at 127.4.5.67:50105: session id = 5ad3a6abdbc846c1a6ed42269a100dd3-0b7371b6f21b4786b16678e0798be3a3
I20240503 02:11:58.682109 6566 tablet_copy_source_session.cc:205] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.682554 8661 tablet_copy_client.cc:308] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:39365
I20240503 02:11:58.693689 6566 tablet_copy_service.cc:140] P d305ba2a2a4640f7997613a7b8884dee: Received BeginTabletCopySession request for tablet 1c6cdade9a8241e4a0808deded2933f9 from peer 329369f9124942438b3176e3b0bacf98 ({username='slave'} at 127.4.5.70:57133)
I20240503 02:11:58.693948 6566 tablet_copy_service.cc:161] P d305ba2a2a4640f7997613a7b8884dee: Beginning new tablet copy session on tablet 1c6cdade9a8241e4a0808deded2933f9 from peer 329369f9124942438b3176e3b0bacf98 at {username='slave'} at 127.4.5.70:57133: session id = 329369f9124942438b3176e3b0bacf98-1c6cdade9a8241e4a0808deded2933f9
I20240503 02:11:58.694450 8660 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0b7371b6f21b4786b16678e0798be3a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.697343 6566 tablet_copy_source_session.cc:205] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:11:58.699231 8661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c6cdade9a8241e4a0808deded2933f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:11:58.702373 8630 tablet_bootstrap.cc:492] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.703140 8630 tablet_bootstrap.cc:492] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: Bootstrap complete.
I20240503 02:11:58.703758 8630 ts_tablet_manager.cc:1367] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: Time spent bootstrapping tablet: real 0.145s user 0.100s sys 0.023s
I20240503 02:11:58.708024 8660 tablet_copy_client.cc:783] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.708359 8660 tablet_copy_client.cc:647] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.711516 8661 tablet_copy_client.cc:783] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.711783 8660 tablet_copy_client.cc:523] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.712175 8661 tablet_copy_client.cc:647] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.712203 8630 raft_consensus.cc:348] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } }
I20240503 02:11:58.712879 8630 raft_consensus.cc:729] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3fafd8e3e974740b2525f2e89d7cca5, State: Initialized, Role: LEARNER
I20240503 02:11:58.714232 8630 consensus_queue.cc:260] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } }
I20240503 02:11:58.715952 8661 tablet_copy_client.cc:523] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.716027 8660 tablet_bootstrap.cc:492] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: Bootstrap starting.
I20240503 02:11:58.718400 8633 log.cc:824] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: Log is configured to *not* fsync() on all Append() calls
I20240503 02:11:58.723125 8630 ts_tablet_manager.cc:1398] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5: Time spent starting tablet: real 0.019s user 0.007s sys 0.010s
I20240503 02:11:58.724385 6697 tablet_copy_service.cc:342] P 4e21bb5794994a489ebd09ee30b6df70: Request end of tablet copy session e3fafd8e3e974740b2525f2e89d7cca5-38eaf65d847745e49865a6cff720cd7e received from {username='slave'} at 127.4.5.69:53415
I20240503 02:11:58.724689 6697 tablet_copy_service.cc:433] P 4e21bb5794994a489ebd09ee30b6df70: ending tablet copy session e3fafd8e3e974740b2525f2e89d7cca5-38eaf65d847745e49865a6cff720cd7e on tablet 38eaf65d847745e49865a6cff720cd7e with peer e3fafd8e3e974740b2525f2e89d7cca5
I20240503 02:11:58.728089 8663 tablet_copy_client.cc:783] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: tablet copy: Starting download of 0 data blocks...
I20240503 02:11:58.728933 8663 tablet_copy_client.cc:647] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: tablet copy: Starting download of 1 WAL segments...
I20240503 02:11:58.731040 8663 tablet_copy_client.cc:523] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:11:58.732659 8661 tablet_bootstrap.cc:492] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: Bootstrap starting.
I20240503 02:11:58.734948 8663 tablet_bootstrap.cc:492] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: Bootstrap starting.
I20240503 02:11:58.735803 8609 tablet_bootstrap.cc:492] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.740613 8609 tablet_bootstrap.cc:492] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: Bootstrap complete.
I20240503 02:11:58.741292 8609 ts_tablet_manager.cc:1367] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: Time spent bootstrapping tablet: real 0.264s user 0.086s sys 0.009s
I20240503 02:11:58.763456 8609 raft_consensus.cc:348] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } }
I20240503 02:11:58.764174 8609 raft_consensus.cc:729] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4bf9c9a4cde949dbabf5144188edbf5b, State: Initialized, Role: LEARNER
I20240503 02:11:58.765489 8609 consensus_queue.cc:260] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } }
I20240503 02:11:58.780913 8609 ts_tablet_manager.cc:1398] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b: Time spent starting tablet: real 0.039s user 0.016s sys 0.000s
I20240503 02:11:58.784380 6566 tablet_copy_service.cc:342] P d305ba2a2a4640f7997613a7b8884dee: Request end of tablet copy session 4bf9c9a4cde949dbabf5144188edbf5b-ff6cd0b970c5434bac564d6c1b2b20e1 received from {username='slave'} at 127.4.5.71:37275
I20240503 02:11:58.784687 6566 tablet_copy_service.cc:433] P d305ba2a2a4640f7997613a7b8884dee: ending tablet copy session 4bf9c9a4cde949dbabf5144188edbf5b-ff6cd0b970c5434bac564d6c1b2b20e1 on tablet ff6cd0b970c5434bac564d6c1b2b20e1 with peer 4bf9c9a4cde949dbabf5144188edbf5b
I20240503 02:11:58.786798 7941 raft_consensus.cc:1203] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:58.800935 8607 tablet_bootstrap.cc:492] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.801986 8607 tablet_bootstrap.cc:492] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: Bootstrap complete.
I20240503 02:11:58.803316 8607 ts_tablet_manager.cc:1367] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: Time spent bootstrapping tablet: real 0.257s user 0.083s sys 0.007s
I20240503 02:11:58.817332 8607 raft_consensus.cc:348] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } }
I20240503 02:11:58.821056 8607 raft_consensus.cc:729] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 329369f9124942438b3176e3b0bacf98, State: Initialized, Role: LEARNER
I20240503 02:11:58.822712 8607 consensus_queue.cc:260] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } }
I20240503 02:11:58.823338 8646 tablet_bootstrap.cc:492] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.823889 8646 tablet_bootstrap.cc:492] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: Bootstrap complete.
I20240503 02:11:58.824259 8646 ts_tablet_manager.cc:1367] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: Time spent bootstrapping tablet: real 0.183s user 0.047s sys 0.009s
I20240503 02:11:58.829981 8607 ts_tablet_manager.cc:1398] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98: Time spent starting tablet: real 0.026s user 0.009s sys 0.003s
I20240503 02:11:58.842335 6697 tablet_copy_service.cc:342] P 4e21bb5794994a489ebd09ee30b6df70: Request end of tablet copy session 329369f9124942438b3176e3b0bacf98-5c12714b4aab448e85708644e84e807c received from {username='slave'} at 127.4.5.70:37783
I20240503 02:11:58.842283 8646 raft_consensus.cc:348] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } }
I20240503 02:11:58.842550 6697 tablet_copy_service.cc:433] P 4e21bb5794994a489ebd09ee30b6df70: ending tablet copy session 329369f9124942438b3176e3b0bacf98-5c12714b4aab448e85708644e84e807c on tablet 5c12714b4aab448e85708644e84e807c with peer 329369f9124942438b3176e3b0bacf98
I20240503 02:11:58.842636 8646 raft_consensus.cc:729] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4bf9c9a4cde949dbabf5144188edbf5b, State: Initialized, Role: LEARNER
I20240503 02:11:58.842914 8646 consensus_queue.cc:260] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: true } }
I20240503 02:11:58.843720 8646 ts_tablet_manager.cc:1398] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b: Time spent starting tablet: real 0.019s user 0.006s sys 0.000s
I20240503 02:11:58.845430 6697 tablet_copy_service.cc:342] P 4e21bb5794994a489ebd09ee30b6df70: Request end of tablet copy session 4bf9c9a4cde949dbabf5144188edbf5b-c041465d7d0f44f2af2beff9ed50ab96 received from {username='slave'} at 127.4.5.71:56097
I20240503 02:11:58.845633 6697 tablet_copy_service.cc:433] P 4e21bb5794994a489ebd09ee30b6df70: ending tablet copy session 4bf9c9a4cde949dbabf5144188edbf5b-c041465d7d0f44f2af2beff9ed50ab96 on tablet c041465d7d0f44f2af2beff9ed50ab96 with peer 4bf9c9a4cde949dbabf5144188edbf5b
I20240503 02:11:58.859903 8663 tablet_bootstrap.cc:492] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.861454 8663 tablet_bootstrap.cc:492] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: Bootstrap complete.
I20240503 02:11:58.861788 8663 ts_tablet_manager.cc:1367] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: Time spent bootstrapping tablet: real 0.127s user 0.041s sys 0.044s
I20240503 02:11:58.863099 8663 raft_consensus.cc:348] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } }
I20240503 02:11:58.863375 8663 raft_consensus.cc:729] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3fafd8e3e974740b2525f2e89d7cca5, State: Initialized, Role: LEARNER
I20240503 02:11:58.863598 8663 consensus_queue.cc:260] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: true } }
I20240503 02:11:58.864527 8633 tablet_bootstrap.cc:492] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.865212 8633 tablet_bootstrap.cc:492] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: Bootstrap complete.
I20240503 02:11:58.865854 8633 ts_tablet_manager.cc:1367] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: Time spent bootstrapping tablet: real 0.258s user 0.080s sys 0.004s
I20240503 02:11:58.868237 8663 ts_tablet_manager.cc:1398] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:11:58.879267 8217 raft_consensus.cc:1203] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:58.881713 8661 tablet_bootstrap.cc:492] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.881937 6697 tablet_copy_service.cc:342] P 4e21bb5794994a489ebd09ee30b6df70: Request end of tablet copy session e3fafd8e3e974740b2525f2e89d7cca5-9bd51c9555b648d1baf4d1dcddc98fd6 received from {username='slave'} at 127.4.5.69:53415
I20240503 02:11:58.882113 6697 tablet_copy_service.cc:433] P 4e21bb5794994a489ebd09ee30b6df70: ending tablet copy session e3fafd8e3e974740b2525f2e89d7cca5-9bd51c9555b648d1baf4d1dcddc98fd6 on tablet 9bd51c9555b648d1baf4d1dcddc98fd6 with peer e3fafd8e3e974740b2525f2e89d7cca5
I20240503 02:11:58.880715 8633 raft_consensus.cc:348] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } }
I20240503 02:11:58.882843 8633 raft_consensus.cc:729] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ad3a6abdbc846c1a6ed42269a100dd3, State: Initialized, Role: LEARNER
I20240503 02:11:58.883769 8633 consensus_queue.cc:260] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } }
I20240503 02:11:58.884549 8661 tablet_bootstrap.cc:492] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: Bootstrap complete.
I20240503 02:11:58.884979 8661 ts_tablet_manager.cc:1367] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: Time spent bootstrapping tablet: real 0.152s user 0.071s sys 0.012s
I20240503 02:11:58.886459 8661 raft_consensus.cc:348] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } }
I20240503 02:11:58.886765 8661 raft_consensus.cc:729] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 329369f9124942438b3176e3b0bacf98, State: Initialized, Role: LEARNER
I20240503 02:11:58.887190 8661 consensus_queue.cc:260] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: true } }
I20240503 02:11:58.888011 8661 ts_tablet_manager.cc:1398] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 02:11:58.889147 6566 tablet_copy_service.cc:342] P d305ba2a2a4640f7997613a7b8884dee: Request end of tablet copy session 329369f9124942438b3176e3b0bacf98-1c6cdade9a8241e4a0808deded2933f9 received from {username='slave'} at 127.4.5.70:57133
I20240503 02:11:58.889356 6566 tablet_copy_service.cc:433] P d305ba2a2a4640f7997613a7b8884dee: ending tablet copy session 329369f9124942438b3176e3b0bacf98-1c6cdade9a8241e4a0808deded2933f9 on tablet 1c6cdade9a8241e4a0808deded2933f9 with peer 329369f9124942438b3176e3b0bacf98
I20240503 02:11:58.898046 8633 ts_tablet_manager.cc:1398] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3: Time spent starting tablet: real 0.032s user 0.012s sys 0.001s
I20240503 02:11:58.899447 6566 tablet_copy_service.cc:342] P d305ba2a2a4640f7997613a7b8884dee: Request end of tablet copy session 5ad3a6abdbc846c1a6ed42269a100dd3-fe9f9a1a82914859bd08dd9f41de618a received from {username='slave'} at 127.4.5.67:50105
I20240503 02:11:58.899658 6566 tablet_copy_service.cc:433] P d305ba2a2a4640f7997613a7b8884dee: ending tablet copy session 5ad3a6abdbc846c1a6ed42269a100dd3-fe9f9a1a82914859bd08dd9f41de618a on tablet fe9f9a1a82914859bd08dd9f41de618a with peer 5ad3a6abdbc846c1a6ed42269a100dd3
I20240503 02:11:58.902344 8660 tablet_bootstrap.cc:492] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:11:58.903162 8660 tablet_bootstrap.cc:492] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: Bootstrap complete.
I20240503 02:11:58.903541 8660 ts_tablet_manager.cc:1367] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: Time spent bootstrapping tablet: real 0.188s user 0.073s sys 0.000s
I20240503 02:11:58.904753 8660 raft_consensus.cc:348] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } }
I20240503 02:11:58.905027 8660 raft_consensus.cc:729] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 5ad3a6abdbc846c1a6ed42269a100dd3, State: Initialized, Role: LEARNER
I20240503 02:11:58.905297 8660 consensus_queue.cc:260] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: true } }
I20240503 02:11:58.905978 8660 ts_tablet_manager.cc:1398] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:11:58.906685 6566 tablet_copy_service.cc:342] P d305ba2a2a4640f7997613a7b8884dee: Request end of tablet copy session 5ad3a6abdbc846c1a6ed42269a100dd3-0b7371b6f21b4786b16678e0798be3a3 received from {username='slave'} at 127.4.5.67:50105
I20240503 02:11:58.906847 6566 tablet_copy_service.cc:433] P d305ba2a2a4640f7997613a7b8884dee: ending tablet copy session 5ad3a6abdbc846c1a6ed42269a100dd3-0b7371b6f21b4786b16678e0798be3a3 on tablet 0b7371b6f21b4786b16678e0798be3a3 with peer 5ad3a6abdbc846c1a6ed42269a100dd3
I20240503 02:11:58.913165 8084 raft_consensus.cc:1203] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:59.003584 8351 raft_consensus.cc:1203] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:59.020960 7803 raft_consensus.cc:1203] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:59.089396 8084 raft_consensus.cc:1203] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:59.169728 8351 raft_consensus.cc:1203] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:59.177843 7803 raft_consensus.cc:1203] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:59.261977 8217 raft_consensus.cc:1203] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:11:59.263976 8659 raft_consensus.cc:1050] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: attempting to promote NON_VOTER 6f857824c6df4cb3a0eb69a4b66810f9 to VOTER
I20240503 02:11:59.265165 8659 consensus_queue.cc:237] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } }
I20240503 02:11:59.267493 7941 raft_consensus.cc:1261] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 LEARNER]: Refusing update from remote peer d305ba2a2a4640f7997613a7b8884dee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.268344 8491 consensus_queue.cc:1035] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.271736 8659 raft_consensus.cc:2932] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } } }
I20240503 02:11:59.272495 7941 raft_consensus.cc:2932] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } } }
I20240503 02:11:59.278105 6401 catalog_manager.cc:5484] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 reported cstate change: config changed from index 4 to 5, 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } } }
I20240503 02:11:59.401118 8700 raft_consensus.cc:1050] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: attempting to promote NON_VOTER e3fafd8e3e974740b2525f2e89d7cca5 to VOTER
I20240503 02:11:59.402225 8700 consensus_queue.cc:237] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:11:59.404961 8084 raft_consensus.cc:1261] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 LEARNER]: Refusing update from remote peer 4e21bb5794994a489ebd09ee30b6df70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.405774 8539 consensus_queue.cc:1035] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.410523 8700 raft_consensus.cc:2932] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } } }
I20240503 02:11:59.411581 8084 raft_consensus.cc:2932] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } } }
I20240503 02:11:59.416006 6400 catalog_manager.cc:5484] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index 4 to 5, e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.436918 8703 raft_consensus.cc:1050] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: attempting to promote NON_VOTER 329369f9124942438b3176e3b0bacf98 to VOTER
I20240503 02:11:59.437765 8703 consensus_queue.cc:237] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:11:59.440233 8217 raft_consensus.cc:1261] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 1 LEARNER]: Refusing update from remote peer 4e21bb5794994a489ebd09ee30b6df70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.441025 8703 consensus_queue.cc:1035] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.445474 8700 raft_consensus.cc:2932] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 329369f9124942438b3176e3b0bacf98 (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } } }
I20240503 02:11:59.446426 8217 raft_consensus.cc:2932] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 329369f9124942438b3176e3b0bacf98 (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } } }
I20240503 02:11:59.452472 6401 catalog_manager.cc:5484] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index 4 to 5, 329369f9124942438b3176e3b0bacf98 (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.464200 8659 raft_consensus.cc:1050] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: attempting to promote NON_VOTER 5ad3a6abdbc846c1a6ed42269a100dd3 to VOTER
I20240503 02:11:59.465049 8659 consensus_queue.cc:237] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:11:59.467931 7803 raft_consensus.cc:1261] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 LEARNER]: Refusing update from remote peer d305ba2a2a4640f7997613a7b8884dee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.468917 8491 consensus_queue.cc:1035] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.474726 8719 raft_consensus.cc:2932] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } } }
I20240503 02:11:59.477560 7803 raft_consensus.cc:2932] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } } }
I20240503 02:11:59.482070 8700 raft_consensus.cc:1050] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: attempting to promote NON_VOTER e3fafd8e3e974740b2525f2e89d7cca5 to VOTER
I20240503 02:11:59.482411 6400 catalog_manager.cc:5484] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index 4 to 5, 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.482885 8700 consensus_queue.cc:237] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:11:59.485361 8084 raft_consensus.cc:1261] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 LEARNER]: Refusing update from remote peer 4e21bb5794994a489ebd09ee30b6df70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.486523 8539 consensus_queue.cc:1035] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.490345 8703 raft_consensus.cc:2932] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } } }
I20240503 02:11:59.495566 8084 raft_consensus.cc:2932] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } } }
I20240503 02:11:59.496373 6401 catalog_manager.cc:5484] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index 4 to 5, e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.514235 6677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c12714b4aab448e85708644e84e807c"
dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46614
I20240503 02:11:59.514572 6677 raft_consensus.cc:595] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.547237 6546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "fe9f9a1a82914859bd08dd9f41de618a"
dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36578
I20240503 02:11:59.547580 6546 raft_consensus.cc:595] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.578979 6546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "587895f39d8b47858e704e8ccb274b89"
dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36590
I20240503 02:11:59.580083 6546 raft_consensus.cc:595] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.597674 8719 raft_consensus.cc:1050] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: attempting to promote NON_VOTER 5ad3a6abdbc846c1a6ed42269a100dd3 to VOTER
I20240503 02:11:59.598570 8719 consensus_queue.cc:237] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:11:59.600837 6677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "38eaf65d847745e49865a6cff720cd7e"
dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46618
I20240503 02:11:59.601184 6677 raft_consensus.cc:595] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.610819 8539 raft_consensus.cc:1050] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: attempting to promote NON_VOTER 4bf9c9a4cde949dbabf5144188edbf5b to VOTER
I20240503 02:11:59.611557 8539 consensus_queue.cc:237] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:11:59.617089 8351 raft_consensus.cc:1261] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 LEARNER]: Refusing update from remote peer 4e21bb5794994a489ebd09ee30b6df70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.617457 7803 raft_consensus.cc:1261] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 LEARNER]: Refusing update from remote peer d305ba2a2a4640f7997613a7b8884dee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.617902 8703 consensus_queue.cc:1035] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.618789 8491 consensus_queue.cc:1035] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.623530 8539 raft_consensus.cc:2932] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } } }
I20240503 02:11:59.624583 8351 raft_consensus.cc:2932] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } } }
I20240503 02:11:59.630364 6400 catalog_manager.cc:5484] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index 4 to 5, 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.637912 8719 raft_consensus.cc:2932] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } } }
I20240503 02:11:59.638813 7803 raft_consensus.cc:2932] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } } }
I20240503 02:11:59.642915 6401 catalog_manager.cc:5484] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index 4 to 5, 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.780475 8491 raft_consensus.cc:979] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: : Instructing follower 6f857824c6df4cb3a0eb69a4b66810f9 to start an election
I20240503 02:11:59.780712 8491 raft_consensus.cc:1067] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Signalling peer 6f857824c6df4cb3a0eb69a4b66810f9 to start an election
I20240503 02:11:59.781821 7941 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "587895f39d8b47858e704e8ccb274b89"
dest_uuid: "6f857824c6df4cb3a0eb69a4b66810f9"
from {username='slave'} at 127.4.5.65:55091
I20240503 02:11:59.782240 7941 raft_consensus.cc:482] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:59.782397 7941 raft_consensus.cc:3037] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:59.783686 8491 raft_consensus.cc:1050] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: attempting to promote NON_VOTER 4bf9c9a4cde949dbabf5144188edbf5b to VOTER
I20240503 02:11:59.784518 8491 consensus_queue.cc:237] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:11:59.785233 7941 raft_consensus.cc:504] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } }
I20240503 02:11:59.786636 7941 leader_election.cc:288] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [CANDIDATE]: Term 2 election: Requested vote from peers d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:11:59.787351 8351 raft_consensus.cc:1261] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 LEARNER]: Refusing update from remote peer d305ba2a2a4640f7997613a7b8884dee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.787325 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587895f39d8b47858e704e8ccb274b89" candidate_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
I20240503 02:11:59.787691 6546 raft_consensus.cc:3032] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:59.787837 6546 raft_consensus.cc:729] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:59.787984 8491 consensus_queue.cc:1035] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.788166 6546 consensus_queue.cc:260] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } }
I20240503 02:11:59.788597 6546 raft_consensus.cc:3037] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:59.791107 8719 raft_consensus.cc:2932] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } } }
I20240503 02:11:59.791792 6546 raft_consensus.cc:2454] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6f857824c6df4cb3a0eb69a4b66810f9 in term 2.
I20240503 02:11:59.791926 8351 raft_consensus.cc:2932] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } } }
I20240503 02:11:59.792536 7872 leader_election.cc:302] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 6f857824c6df4cb3a0eb69a4b66810f9, d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:59.793335 8854 raft_consensus.cc:2781] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:59.793853 8854 raft_consensus.cc:686] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 2 LEADER]: Becoming Leader. State: Replica: 6f857824c6df4cb3a0eb69a4b66810f9, State: Running, Role: LEADER
I20240503 02:11:59.794297 8854 consensus_queue.cc:237] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } }
I20240503 02:11:59.797410 6401 catalog_manager.cc:5484] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b reported cstate change: config changed from index 4 to 5, 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } } }
I20240503 02:11:59.799094 6399 catalog_manager.cc:5484] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 reported cstate change: term changed from 1 to 2, leader changed from d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) to 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68). New cstate: current_term: 2 leader_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:11:59.810771 6391 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 587895f39d8b47858e704e8ccb274b89 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader 6f857824c6df4cb3a0eb69a4b66810f9 (127.4.5.68:42071) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 02:11:59.837999 8700 raft_consensus.cc:979] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: : Instructing follower e3fafd8e3e974740b2525f2e89d7cca5 to start an election
I20240503 02:11:59.838261 8539 raft_consensus.cc:1067] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Signalling peer e3fafd8e3e974740b2525f2e89d7cca5 to start an election
I20240503 02:11:59.839417 8084 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "38eaf65d847745e49865a6cff720cd7e"
dest_uuid: "e3fafd8e3e974740b2525f2e89d7cca5"
from {username='slave'} at 127.4.5.66:42827
I20240503 02:11:59.839814 8084 raft_consensus.cc:482] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:59.840063 8084 raft_consensus.cc:3037] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:59.843029 8084 raft_consensus.cc:504] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:11:59.845081 8084 leader_election.cc:288] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [CANDIDATE]: Term 2 election: Requested vote from peers 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:11:59.845911 6677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eaf65d847745e49865a6cff720cd7e" candidate_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
I20240503 02:11:59.846285 6677 raft_consensus.cc:3032] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:59.846412 6677 raft_consensus.cc:729] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:11:59.846793 6677 consensus_queue.cc:260] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:11:59.847131 6677 raft_consensus.cc:3037] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:59.848685 8719 raft_consensus.cc:1050] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: attempting to promote NON_VOTER 329369f9124942438b3176e3b0bacf98 to VOTER
I20240503 02:11:59.849491 8719 consensus_queue.cc:237] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:11:59.851222 6677 raft_consensus.cc:2454] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fafd8e3e974740b2525f2e89d7cca5 in term 2.
I20240503 02:11:59.851967 8217 raft_consensus.cc:1261] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 1 LEARNER]: Refusing update from remote peer d305ba2a2a4640f7997613a7b8884dee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:11:59.852034 8015 leader_election.cc:302] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70, e3fafd8e3e974740b2525f2e89d7cca5; no voters:
I20240503 02:11:59.852860 8685 raft_consensus.cc:2781] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:59.852845 8491 consensus_queue.cc:1035] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:11:59.853335 8685 raft_consensus.cc:686] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 2 LEADER]: Becoming Leader. State: Replica: e3fafd8e3e974740b2525f2e89d7cca5, State: Running, Role: LEADER
I20240503 02:11:59.853900 8685 consensus_queue.cc:237] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:11:59.858186 8719 raft_consensus.cc:2932] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 329369f9124942438b3176e3b0bacf98 (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } } }
I20240503 02:11:59.858767 6400 catalog_manager.cc:5484] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 reported cstate change: term changed from 1 to 2, leader changed from 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) to e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69). New cstate: current_term: 2 leader_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.859170 8217 raft_consensus.cc:2932] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 329369f9124942438b3176e3b0bacf98 (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } } }
I20240503 02:11:59.864989 6399 catalog_manager.cc:5484] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index 4 to 5, 329369f9124942438b3176e3b0bacf98 (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.868840 6677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "c041465d7d0f44f2af2beff9ed50ab96"
dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46648
I20240503 02:11:59.869141 6677 raft_consensus.cc:595] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.893002 6677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bd51c9555b648d1baf4d1dcddc98fd6"
dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46666
I20240503 02:11:59.893272 6677 raft_consensus.cc:595] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.911602 6677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "5c12714b4aab448e85708644e84e807c"
dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46690
I20240503 02:11:59.911839 6677 raft_consensus.cc:595] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.932040 8659 raft_consensus.cc:979] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: : Instructing follower 5ad3a6abdbc846c1a6ed42269a100dd3 to start an election
I20240503 02:11:59.932040 6546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "ff6cd0b970c5434bac564d6c1b2b20e1"
dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36620
I20240503 02:11:59.932312 8659 raft_consensus.cc:1067] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Signalling peer 5ad3a6abdbc846c1a6ed42269a100dd3 to start an election
I20240503 02:11:59.932380 6546 raft_consensus.cc:595] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Received request to transfer leadership
I20240503 02:11:59.933207 7803 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "fe9f9a1a82914859bd08dd9f41de618a"
dest_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3"
from {username='slave'} at 127.4.5.65:47157
I20240503 02:11:59.933566 7803 raft_consensus.cc:482] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:11:59.933791 7803 raft_consensus.cc:3037] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:59.936609 7803 raft_consensus.cc:504] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:11:59.938831 7803 leader_election.cc:288] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [CANDIDATE]: Term 2 election: Requested vote from peers d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:11:59.939344 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fe9f9a1a82914859bd08dd9f41de618a" candidate_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
I20240503 02:11:59.939635 6546 raft_consensus.cc:3032] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:11:59.939756 6546 raft_consensus.cc:729] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:11:59.940088 6546 consensus_queue.cc:260] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:11:59.940392 6546 raft_consensus.cc:3037] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:11:59.943570 6546 raft_consensus.cc:2454] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad3a6abdbc846c1a6ed42269a100dd3 in term 2.
I20240503 02:11:59.944259 7736 leader_election.cc:302] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5ad3a6abdbc846c1a6ed42269a100dd3, d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:11:59.944772 8702 raft_consensus.cc:2781] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:11:59.945312 8702 raft_consensus.cc:686] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 5ad3a6abdbc846c1a6ed42269a100dd3, State: Running, Role: LEADER
I20240503 02:11:59.945948 8702 consensus_queue.cc:237] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:11:59.950762 6399 catalog_manager.cc:5484] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 reported cstate change: term changed from 1 to 2, leader changed from d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) to 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67). New cstate: current_term: 2 leader_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:11:59.994303 6546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0b7371b6f21b4786b16678e0798be3a3"
dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36652
I20240503 02:11:59.994609 6546 raft_consensus.cc:595] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:00.071251 8700 raft_consensus.cc:979] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: : Instructing follower 329369f9124942438b3176e3b0bacf98 to start an election
I20240503 02:12:00.071476 8700 raft_consensus.cc:1067] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Signalling peer 329369f9124942438b3176e3b0bacf98 to start an election
I20240503 02:12:00.072315 8217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "5c12714b4aab448e85708644e84e807c"
dest_uuid: "329369f9124942438b3176e3b0bacf98"
from {username='slave'} at 127.4.5.66:39977
I20240503 02:12:00.072695 8217 raft_consensus.cc:482] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:00.072898 8217 raft_consensus.cc:3037] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.075364 8217 raft_consensus.cc:504] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:12:00.077066 8217 leader_election.cc:288] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [CANDIDATE]: Term 2 election: Requested vote from peers 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:12:00.077791 6677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c12714b4aab448e85708644e84e807c" candidate_uuid: "329369f9124942438b3176e3b0bacf98" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
I20240503 02:12:00.078023 6677 raft_consensus.cc:3032] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:00.078135 6677 raft_consensus.cc:729] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:12:00.078442 6677 consensus_queue.cc:260] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:12:00.078770 6677 raft_consensus.cc:3037] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.080987 6677 raft_consensus.cc:2454] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 329369f9124942438b3176e3b0bacf98 in term 2.
I20240503 02:12:00.081676 8148 leader_election.cc:302] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 329369f9124942438b3176e3b0bacf98, 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:12:00.082185 8697 raft_consensus.cc:2781] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:00.082674 8697 raft_consensus.cc:686] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 2 LEADER]: Becoming Leader. State: Replica: 329369f9124942438b3176e3b0bacf98, State: Running, Role: LEADER
I20240503 02:12:00.083325 8697 consensus_queue.cc:237] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:12:00.087567 6399 catalog_manager.cc:5484] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 reported cstate change: term changed from 1 to 2, leader changed from 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) to 329369f9124942438b3176e3b0bacf98 (127.4.5.70). New cstate: current_term: 2 leader_uuid: "329369f9124942438b3176e3b0bacf98" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.099243 8719 raft_consensus.cc:979] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: : Instructing follower 5ad3a6abdbc846c1a6ed42269a100dd3 to start an election
I20240503 02:12:00.099476 8659 raft_consensus.cc:1067] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Signalling peer 5ad3a6abdbc846c1a6ed42269a100dd3 to start an election
I20240503 02:12:00.100172 7803 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0b7371b6f21b4786b16678e0798be3a3"
dest_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3"
from {username='slave'} at 127.4.5.65:47157
I20240503 02:12:00.100488 7803 raft_consensus.cc:482] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:00.100625 7803 raft_consensus.cc:3037] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.103796 7803 raft_consensus.cc:504] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:12:00.104413 7803 leader_election.cc:288] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [CANDIDATE]: Term 2 election: Requested vote from peers d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:12:00.105119 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0b7371b6f21b4786b16678e0798be3a3" candidate_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
I20240503 02:12:00.105384 6546 raft_consensus.cc:3032] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:00.105473 6546 raft_consensus.cc:729] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:12:00.105791 6546 consensus_queue.cc:260] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:12:00.105988 6546 raft_consensus.cc:3037] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.108140 6546 raft_consensus.cc:2454] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5ad3a6abdbc846c1a6ed42269a100dd3 in term 2.
I20240503 02:12:00.108788 7736 leader_election.cc:302] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 5ad3a6abdbc846c1a6ed42269a100dd3, d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:12:00.109189 8702 raft_consensus.cc:2781] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:00.109448 8702 raft_consensus.cc:686] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 2 LEADER]: Becoming Leader. State: Replica: 5ad3a6abdbc846c1a6ed42269a100dd3, State: Running, Role: LEADER
I20240503 02:12:00.109772 8702 consensus_queue.cc:237] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:12:00.112650 6399 catalog_manager.cc:5484] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 reported cstate change: term changed from 1 to 2, leader changed from d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) to 5ad3a6abdbc846c1a6ed42269a100dd3 (127.4.5.67). New cstate: current_term: 2 leader_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.212154 8539 raft_consensus.cc:979] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: : Instructing follower 4bf9c9a4cde949dbabf5144188edbf5b to start an election
I20240503 02:12:00.212394 8539 raft_consensus.cc:1067] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Signalling peer 4bf9c9a4cde949dbabf5144188edbf5b to start an election
I20240503 02:12:00.213275 8351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "c041465d7d0f44f2af2beff9ed50ab96"
dest_uuid: "4bf9c9a4cde949dbabf5144188edbf5b"
from {username='slave'} at 127.4.5.66:50399
I20240503 02:12:00.213603 8351 raft_consensus.cc:482] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:00.213804 8351 raft_consensus.cc:3037] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.216115 6546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1c6cdade9a8241e4a0808deded2933f9"
dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:36680
I20240503 02:12:00.216360 6546 raft_consensus.cc:595] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:00.217770 8351 raft_consensus.cc:504] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:12:00.220336 8351 leader_election.cc:288] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [CANDIDATE]: Term 2 election: Requested vote from peers 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:12:00.221074 6677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c041465d7d0f44f2af2beff9ed50ab96" candidate_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
I20240503 02:12:00.221350 6677 raft_consensus.cc:3032] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:00.221453 6677 raft_consensus.cc:729] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:12:00.221828 6677 consensus_queue.cc:260] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:12:00.222179 6677 raft_consensus.cc:3037] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.224977 6677 raft_consensus.cc:2454] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bf9c9a4cde949dbabf5144188edbf5b in term 2.
I20240503 02:12:00.225656 8283 leader_election.cc:302] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4bf9c9a4cde949dbabf5144188edbf5b, 4e21bb5794994a489ebd09ee30b6df70; no voters:
I20240503 02:12:00.226204 8692 raft_consensus.cc:2781] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:00.226727 8692 raft_consensus.cc:686] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 2 LEADER]: Becoming Leader. State: Replica: 4bf9c9a4cde949dbabf5144188edbf5b, State: Running, Role: LEADER
I20240503 02:12:00.227373 8692 consensus_queue.cc:237] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:12:00.231878 6399 catalog_manager.cc:5484] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b reported cstate change: term changed from 1 to 2, leader changed from 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) to 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71). New cstate: current_term: 2 leader_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.233690 8719 raft_consensus.cc:979] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: : Instructing follower 4bf9c9a4cde949dbabf5144188edbf5b to start an election
I20240503 02:12:00.233891 8719 raft_consensus.cc:1067] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Signalling peer 4bf9c9a4cde949dbabf5144188edbf5b to start an election
I20240503 02:12:00.234544 8351 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "ff6cd0b970c5434bac564d6c1b2b20e1"
dest_uuid: "4bf9c9a4cde949dbabf5144188edbf5b"
from {username='slave'} at 127.4.5.65:33655
I20240503 02:12:00.234777 8351 raft_consensus.cc:482] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:00.234895 8351 raft_consensus.cc:3037] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.237432 8351 raft_consensus.cc:504] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:12:00.238729 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ff6cd0b970c5434bac564d6c1b2b20e1" candidate_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
I20240503 02:12:00.238955 6546 raft_consensus.cc:3032] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:00.239018 6546 raft_consensus.cc:729] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:12:00.239257 6546 consensus_queue.cc:260] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:12:00.239442 6546 raft_consensus.cc:3037] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.240293 8351 leader_election.cc:288] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [CANDIDATE]: Term 2 election: Requested vote from peers d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:12:00.243628 6546 raft_consensus.cc:2454] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bf9c9a4cde949dbabf5144188edbf5b in term 2.
I20240503 02:12:00.245168 8284 leader_election.cc:302] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4bf9c9a4cde949dbabf5144188edbf5b, d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:12:00.245494 8692 raft_consensus.cc:2781] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:00.245718 8692 raft_consensus.cc:686] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 2 LEADER]: Becoming Leader. State: Replica: 4bf9c9a4cde949dbabf5144188edbf5b, State: Running, Role: LEADER
I20240503 02:12:00.246062 8692 consensus_queue.cc:237] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:12:00.249398 6399 catalog_manager.cc:5484] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b reported cstate change: term changed from 1 to 2, leader changed from d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) to 4bf9c9a4cde949dbabf5144188edbf5b (127.4.5.71). New cstate: current_term: 2 leader_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.261075 6677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9bd51c9555b648d1baf4d1dcddc98fd6"
dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:46716
I20240503 02:12:00.261335 6677 raft_consensus.cc:595] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:00.343511 6546 raft_consensus.cc:1261] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Refusing update from remote peer 6f857824c6df4cb3a0eb69a4b66810f9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.344552 8995 consensus_queue.cc:1035] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:00.384932 8719 raft_consensus.cc:979] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: : Instructing follower 329369f9124942438b3176e3b0bacf98 to start an election
I20240503 02:12:00.385145 8719 raft_consensus.cc:1067] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Signalling peer 329369f9124942438b3176e3b0bacf98 to start an election
I20240503 02:12:00.385855 8217 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1c6cdade9a8241e4a0808deded2933f9"
dest_uuid: "329369f9124942438b3176e3b0bacf98"
from {username='slave'} at 127.4.5.65:37033
I20240503 02:12:00.386111 8217 raft_consensus.cc:482] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:00.386236 8217 raft_consensus.cc:3037] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.388414 8217 raft_consensus.cc:504] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:12:00.389122 8217 leader_election.cc:288] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [CANDIDATE]: Term 2 election: Requested vote from peers d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:12:00.389750 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c6cdade9a8241e4a0808deded2933f9" candidate_uuid: "329369f9124942438b3176e3b0bacf98" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
I20240503 02:12:00.389976 6546 raft_consensus.cc:3032] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:00.390080 6546 raft_consensus.cc:729] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:12:00.390388 6546 consensus_queue.cc:260] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:12:00.390759 6546 raft_consensus.cc:3037] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.392695 6677 raft_consensus.cc:1261] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Refusing update from remote peer e3fafd8e3e974740b2525f2e89d7cca5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.393205 6546 raft_consensus.cc:2454] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 329369f9124942438b3176e3b0bacf98 in term 2.
I20240503 02:12:00.393675 8149 leader_election.cc:302] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 329369f9124942438b3176e3b0bacf98, d305ba2a2a4640f7997613a7b8884dee; no voters:
I20240503 02:12:00.393647 9000 consensus_queue.cc:1035] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:00.393993 8697 raft_consensus.cc:2781] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:00.394171 8697 raft_consensus.cc:686] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 2 LEADER]: Becoming Leader. State: Replica: 329369f9124942438b3176e3b0bacf98, State: Running, Role: LEADER
I20240503 02:12:00.394495 8697 consensus_queue.cc:237] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:12:00.397074 6400 catalog_manager.cc:5484] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 reported cstate change: term changed from 1 to 2, leader changed from d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) to 329369f9124942438b3176e3b0bacf98 (127.4.5.70). New cstate: current_term: 2 leader_uuid: "329369f9124942438b3176e3b0bacf98" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.424293 8700 raft_consensus.cc:979] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: : Instructing follower e3fafd8e3e974740b2525f2e89d7cca5 to start an election
I20240503 02:12:00.424537 8539 raft_consensus.cc:1067] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Signalling peer e3fafd8e3e974740b2525f2e89d7cca5 to start an election
I20240503 02:12:00.425290 8083 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9bd51c9555b648d1baf4d1dcddc98fd6"
dest_uuid: "e3fafd8e3e974740b2525f2e89d7cca5"
from {username='slave'} at 127.4.5.66:42827
I20240503 02:12:00.425540 8083 raft_consensus.cc:482] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:00.425701 8083 raft_consensus.cc:3037] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.427954 8083 raft_consensus.cc:504] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:12:00.428612 8083 leader_election.cc:288] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [CANDIDATE]: Term 2 election: Requested vote from peers 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:12:00.429276 6677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bd51c9555b648d1baf4d1dcddc98fd6" candidate_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
I20240503 02:12:00.429517 6677 raft_consensus.cc:3032] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:00.429605 6677 raft_consensus.cc:729] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:12:00.429944 6677 consensus_queue.cc:260] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:12:00.430272 6677 raft_consensus.cc:3037] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:00.433471 6677 raft_consensus.cc:2454] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3fafd8e3e974740b2525f2e89d7cca5 in term 2.
I20240503 02:12:00.433967 8015 leader_election.cc:302] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70, e3fafd8e3e974740b2525f2e89d7cca5; no voters:
I20240503 02:12:00.434341 9004 raft_consensus.cc:2781] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:00.434535 9004 raft_consensus.cc:686] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 2 LEADER]: Becoming Leader. State: Replica: e3fafd8e3e974740b2525f2e89d7cca5, State: Running, Role: LEADER
I20240503 02:12:00.434844 9004 consensus_queue.cc:237] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:12:00.437486 6400 catalog_manager.cc:5484] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 reported cstate change: term changed from 1 to 2, leader changed from 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) to e3fafd8e3e974740b2525f2e89d7cca5 (127.4.5.69). New cstate: current_term: 2 leader_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.478543 7941 consensus_queue.cc:237] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } }
I20240503 02:12:00.479650 8997 raft_consensus.cc:2932] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } } }
I20240503 02:12:00.483188 6391 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 587895f39d8b47858e704e8ccb274b89 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:00.484654 6400 catalog_manager.cc:5484] T 587895f39d8b47858e704e8ccb274b89 P 6f857824c6df4cb3a0eb69a4b66810f9 reported cstate change: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New cstate: current_term: 2 leader_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "6f857824c6df4cb3a0eb69a4b66810f9" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 42071 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.489091 6526 tablet_service.cc:1514] Processing DeleteTablet for tablet 587895f39d8b47858e704e8ccb274b89 with delete_type TABLET_DATA_TOMBSTONED (TS d305ba2a2a4640f7997613a7b8884dee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50604
I20240503 02:12:00.490083 9005 tablet_replica.cc:330] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: stopping tablet replica
I20240503 02:12:00.490574 9005 raft_consensus.cc:2229] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.490868 9005 raft_consensus.cc:2258] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.492606 9005 ts_tablet_manager.cc:1875] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.498051 9005 ts_tablet_manager.cc:1888] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:00.498215 9005 log.cc:1196] T 587895f39d8b47858e704e8ccb274b89 P d305ba2a2a4640f7997613a7b8884dee: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/587895f39d8b47858e704e8ccb274b89
I20240503 02:12:00.499239 6389 catalog_manager.cc:4830] TS d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365): tablet 587895f39d8b47858e704e8ccb274b89 (table rebalance_test_table_0 [id=598e8f31ad644c8fbbaceeffbce04724]) successfully deleted
I20240503 02:12:00.507709 8084 consensus_queue.cc:237] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:12:00.508800 9004 raft_consensus.cc:2932] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } } }
I20240503 02:12:00.512364 6391 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 38eaf65d847745e49865a6cff720cd7e with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:00.514705 6400 catalog_manager.cc:5484] T 38eaf65d847745e49865a6cff720cd7e P e3fafd8e3e974740b2525f2e89d7cca5 reported cstate change: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New cstate: current_term: 2 leader_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.515794 6546 raft_consensus.cc:1261] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Refusing update from remote peer 5ad3a6abdbc846c1a6ed42269a100dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.516584 8702 consensus_queue.cc:1035] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:00.520660 6657 tablet_service.cc:1514] Processing DeleteTablet for tablet 38eaf65d847745e49865a6cff720cd7e with delete_type TABLET_DATA_TOMBSTONED (TS 4e21bb5794994a489ebd09ee30b6df70 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35614
I20240503 02:12:00.527868 9007 tablet_replica.cc:330] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: stopping tablet replica
I20240503 02:12:00.528558 9007 raft_consensus.cc:2229] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.528970 9007 raft_consensus.cc:2258] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.530925 9007 ts_tablet_manager.cc:1875] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.538458 9007 ts_tablet_manager.cc:1888] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:00.538628 9007 log.cc:1196] T 38eaf65d847745e49865a6cff720cd7e P 4e21bb5794994a489ebd09ee30b6df70: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/38eaf65d847745e49865a6cff720cd7e
I20240503 02:12:00.539710 6388 catalog_manager.cc:4830] TS 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573): tablet 38eaf65d847745e49865a6cff720cd7e (table rebalance_test_table_3 [id=107814137f694dceb7cfcaae16c5fd8f]) successfully deleted
I20240503 02:12:00.590533 6546 raft_consensus.cc:1261] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Refusing update from remote peer 5ad3a6abdbc846c1a6ed42269a100dd3: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.591338 9009 consensus_queue.cc:1035] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:00.602842 6677 raft_consensus.cc:1261] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Refusing update from remote peer 329369f9124942438b3176e3b0bacf98: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.603796 8697 consensus_queue.cc:1035] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:00.625005 7803 consensus_queue.cc:237] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:12:00.626207 8702 raft_consensus.cc:2932] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } } }
I20240503 02:12:00.631103 6389 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fe9f9a1a82914859bd08dd9f41de618a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:00.632572 6400 catalog_manager.cc:5484] T fe9f9a1a82914859bd08dd9f41de618a P 5ad3a6abdbc846c1a6ed42269a100dd3 reported cstate change: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New cstate: current_term: 2 leader_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.637025 6526 tablet_service.cc:1514] Processing DeleteTablet for tablet fe9f9a1a82914859bd08dd9f41de618a with delete_type TABLET_DATA_TOMBSTONED (TS d305ba2a2a4640f7997613a7b8884dee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50604
I20240503 02:12:00.637512 9005 tablet_replica.cc:330] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: stopping tablet replica
I20240503 02:12:00.637773 9005 raft_consensus.cc:2229] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.638010 9005 raft_consensus.cc:2258] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.639827 9005 ts_tablet_manager.cc:1875] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.645723 9005 ts_tablet_manager.cc:1888] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:00.645890 9005 log.cc:1196] T fe9f9a1a82914859bd08dd9f41de618a P d305ba2a2a4640f7997613a7b8884dee: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/fe9f9a1a82914859bd08dd9f41de618a
I20240503 02:12:00.646929 6389 catalog_manager.cc:4830] TS d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365): tablet fe9f9a1a82914859bd08dd9f41de618a (table rebalance_test_table_4 [id=c0b61eafd89d498db460c250fdbaa217]) successfully deleted
I20240503 02:12:00.721238 7803 consensus_queue.cc:237] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } }
I20240503 02:12:00.722337 8702 raft_consensus.cc:2932] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } } }
I20240503 02:12:00.726720 6389 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0b7371b6f21b4786b16678e0798be3a3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:00.728120 6399 catalog_manager.cc:5484] T 0b7371b6f21b4786b16678e0798be3a3 P 5ad3a6abdbc846c1a6ed42269a100dd3 reported cstate change: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New cstate: current_term: 2 leader_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "5ad3a6abdbc846c1a6ed42269a100dd3" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41365 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.732677 6526 tablet_service.cc:1514] Processing DeleteTablet for tablet 0b7371b6f21b4786b16678e0798be3a3 with delete_type TABLET_DATA_TOMBSTONED (TS d305ba2a2a4640f7997613a7b8884dee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50604
I20240503 02:12:00.733080 9005 tablet_replica.cc:330] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: stopping tablet replica
I20240503 02:12:00.733300 9005 raft_consensus.cc:2229] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.733521 9005 raft_consensus.cc:2258] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.734812 9005 ts_tablet_manager.cc:1875] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.741029 9005 ts_tablet_manager.cc:1888] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:00.741201 9005 log.cc:1196] T 0b7371b6f21b4786b16678e0798be3a3 P d305ba2a2a4640f7997613a7b8884dee: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/0b7371b6f21b4786b16678e0798be3a3
I20240503 02:12:00.742084 6389 catalog_manager.cc:4830] TS d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365): tablet 0b7371b6f21b4786b16678e0798be3a3 (table rebalance_test_table_1 [id=7dc9b08ad8464ad9b1733bc422522440]) successfully deleted
I20240503 02:12:00.766300 8217 consensus_queue.cc:237] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:12:00.767761 8697 raft_consensus.cc:2932] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } } }
I20240503 02:12:00.771344 6388 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c12714b4aab448e85708644e84e807c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:00.773056 6399 catalog_manager.cc:5484] T 5c12714b4aab448e85708644e84e807c P 329369f9124942438b3176e3b0bacf98 reported cstate change: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New cstate: current_term: 2 leader_uuid: "329369f9124942438b3176e3b0bacf98" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.777338 6657 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c12714b4aab448e85708644e84e807c with delete_type TABLET_DATA_TOMBSTONED (TS 4e21bb5794994a489ebd09ee30b6df70 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35614
I20240503 02:12:00.777782 9007 tablet_replica.cc:330] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: stopping tablet replica
I20240503 02:12:00.777997 9007 raft_consensus.cc:2229] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.778224 9007 raft_consensus.cc:2258] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.779129 9007 ts_tablet_manager.cc:1875] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.783907 9007 ts_tablet_manager.cc:1888] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:00.784035 9007 log.cc:1196] T 5c12714b4aab448e85708644e84e807c P 4e21bb5794994a489ebd09ee30b6df70: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/5c12714b4aab448e85708644e84e807c
I20240503 02:12:00.784932 6388 catalog_manager.cc:4830] TS 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573): tablet 5c12714b4aab448e85708644e84e807c (table rebalance_test_table_1 [id=7dc9b08ad8464ad9b1733bc422522440]) successfully deleted
I20240503 02:12:00.795063 6677 raft_consensus.cc:1261] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Refusing update from remote peer 4bf9c9a4cde949dbabf5144188edbf5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.796043 9113 consensus_queue.cc:1035] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:00.854321 8351 consensus_queue.cc:237] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:12:00.855499 9120 raft_consensus.cc:2932] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } } }
I20240503 02:12:00.858937 6388 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c041465d7d0f44f2af2beff9ed50ab96 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:00.860389 6399 catalog_manager.cc:5484] T c041465d7d0f44f2af2beff9ed50ab96 P 4bf9c9a4cde949dbabf5144188edbf5b reported cstate change: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New cstate: current_term: 2 leader_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.863910 6546 raft_consensus.cc:1261] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Refusing update from remote peer 4bf9c9a4cde949dbabf5144188edbf5b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.864568 6657 tablet_service.cc:1514] Processing DeleteTablet for tablet c041465d7d0f44f2af2beff9ed50ab96 with delete_type TABLET_DATA_TOMBSTONED (TS 4e21bb5794994a489ebd09ee30b6df70 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35614
I20240503 02:12:00.864674 9120 consensus_queue.cc:1035] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:00.865007 9007 tablet_replica.cc:330] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: stopping tablet replica
I20240503 02:12:00.865218 9007 raft_consensus.cc:2229] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.865437 9007 raft_consensus.cc:2258] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.866505 9007 ts_tablet_manager.cc:1875] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.874686 9007 ts_tablet_manager.cc:1888] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:00.874836 9007 log.cc:1196] T c041465d7d0f44f2af2beff9ed50ab96 P 4e21bb5794994a489ebd09ee30b6df70: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/c041465d7d0f44f2af2beff9ed50ab96
I20240503 02:12:00.875603 6388 catalog_manager.cc:4830] TS 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573): tablet c041465d7d0f44f2af2beff9ed50ab96 (table rebalance_test_table_4 [id=c0b61eafd89d498db460c250fdbaa217]) successfully deleted
I20240503 02:12:00.882565 8351 consensus_queue.cc:237] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } }
I20240503 02:12:00.883621 9113 raft_consensus.cc:2932] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } } }
I20240503 02:12:00.886721 6388 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ff6cd0b970c5434bac564d6c1b2b20e1 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:00.888291 6400 catalog_manager.cc:5484] T ff6cd0b970c5434bac564d6c1b2b20e1 P 4bf9c9a4cde949dbabf5144188edbf5b reported cstate change: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New cstate: current_term: 2 leader_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4bf9c9a4cde949dbabf5144188edbf5b" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 37017 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:00.892366 6526 tablet_service.cc:1514] Processing DeleteTablet for tablet ff6cd0b970c5434bac564d6c1b2b20e1 with delete_type TABLET_DATA_TOMBSTONED (TS d305ba2a2a4640f7997613a7b8884dee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50604
I20240503 02:12:00.892840 9005 tablet_replica.cc:330] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: stopping tablet replica
I20240503 02:12:00.893060 9005 raft_consensus.cc:2229] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:00.893282 9005 raft_consensus.cc:2258] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:00.894214 9005 ts_tablet_manager.cc:1875] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:00.895051 6546 raft_consensus.cc:1261] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Refusing update from remote peer 329369f9124942438b3176e3b0bacf98: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:00.895745 8697 consensus_queue.cc:1035] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:00.900509 9005 ts_tablet_manager.cc:1888] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:00.900749 9005 log.cc:1196] T ff6cd0b970c5434bac564d6c1b2b20e1 P d305ba2a2a4640f7997613a7b8884dee: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/ff6cd0b970c5434bac564d6c1b2b20e1
I20240503 02:12:00.901860 6389 catalog_manager.cc:4830] TS d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365): tablet ff6cd0b970c5434bac564d6c1b2b20e1 (table rebalance_test_table_2 [id=efe14e9c57a44f339179a831adbcd921]) successfully deleted
I20240503 02:12:01.017253 6677 raft_consensus.cc:1261] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Refusing update from remote peer e3fafd8e3e974740b2525f2e89d7cca5: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:01.017964 9145 consensus_queue.cc:1035] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:01.027135 8217 consensus_queue.cc:237] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } }
I20240503 02:12:01.028141 9144 raft_consensus.cc:2932] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } } }
I20240503 02:12:01.031262 6388 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1c6cdade9a8241e4a0808deded2933f9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:01.032693 6400 catalog_manager.cc:5484] T 1c6cdade9a8241e4a0808deded2933f9 P 329369f9124942438b3176e3b0bacf98 reported cstate change: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New cstate: current_term: 2 leader_uuid: "329369f9124942438b3176e3b0bacf98" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "329369f9124942438b3176e3b0bacf98" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 40279 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:01.036659 6526 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c6cdade9a8241e4a0808deded2933f9 with delete_type TABLET_DATA_TOMBSTONED (TS d305ba2a2a4640f7997613a7b8884dee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50604
I20240503 02:12:01.037071 9005 tablet_replica.cc:330] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: stopping tablet replica
I20240503 02:12:01.037295 9005 raft_consensus.cc:2229] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:01.037516 9005 raft_consensus.cc:2258] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:01.039353 9005 ts_tablet_manager.cc:1875] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:01.044512 9005 ts_tablet_manager.cc:1888] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:01.044670 9005 log.cc:1196] T 1c6cdade9a8241e4a0808deded2933f9 P d305ba2a2a4640f7997613a7b8884dee: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/1c6cdade9a8241e4a0808deded2933f9
I20240503 02:12:01.045574 6389 catalog_manager.cc:4830] TS d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365): tablet 1c6cdade9a8241e4a0808deded2933f9 (table rebalance_test_table_3 [id=107814137f694dceb7cfcaae16c5fd8f]) successfully deleted
I20240503 02:12:01.060364 8084 consensus_queue.cc:237] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } }
I20240503 02:12:01.061589 9147 raft_consensus.cc:2932] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } } }
I20240503 02:12:01.066202 6391 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9bd51c9555b648d1baf4d1dcddc98fd6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:01.067808 6400 catalog_manager.cc:5484] T 9bd51c9555b648d1baf4d1dcddc98fd6 P e3fafd8e3e974740b2525f2e89d7cca5 reported cstate change: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New cstate: current_term: 2 leader_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "e3fafd8e3e974740b2525f2e89d7cca5" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 35867 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:01.072307 6657 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bd51c9555b648d1baf4d1dcddc98fd6 with delete_type TABLET_DATA_TOMBSTONED (TS 4e21bb5794994a489ebd09ee30b6df70 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35614
I20240503 02:12:01.072803 9007 tablet_replica.cc:330] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: stopping tablet replica
I20240503 02:12:01.073035 9007 raft_consensus.cc:2229] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:01.073276 9007 raft_consensus.cc:2258] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:01.074649 9007 ts_tablet_manager.cc:1875] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:01.080753 9007 ts_tablet_manager.cc:1888] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:01.080926 9007 log.cc:1196] T 9bd51c9555b648d1baf4d1dcddc98fd6 P 4e21bb5794994a489ebd09ee30b6df70: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/9bd51c9555b648d1baf4d1dcddc98fd6
I20240503 02:12:01.081792 6388 catalog_manager.cc:4830] TS 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573): tablet 9bd51c9555b648d1baf4d1dcddc98fd6 (table rebalance_test_table_2 [id=efe14e9c57a44f339179a831adbcd921]) successfully deleted
I20240503 02:12:01.243199 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.244287 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.262555 6657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:01.269917 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.271680 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.302160 6526 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:01.303463 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.506009 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.508280 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.514611 6657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:01.515897 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.533136 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.539886 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:01.541383 6526 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:02.372419 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.376025 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.390475 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.392808 6657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:02.440024 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.453994 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.479599 6526 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:02.703833 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.706574 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.723524 6526 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:02.740051 6657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:02.741358 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.742388 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:02.753234 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:03.069267 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:03.073756 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:03.075165 6657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:03.091197 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:03.106631 6526 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:03.108527 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:03.144654 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
596ca00a45524913952aa8c8b8864eee | 127.4.5.126:42813 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+------------------+-------------------------
builtin_ntp_servers | 127.4.5.84:38437 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
329369f9124942438b3176e3b0bacf98 | 127.4.5.70:40279 | HEALTHY | <none> | 2 | 0
4bf9c9a4cde949dbabf5144188edbf5b | 127.4.5.71:37017 | HEALTHY | <none> | 2 | 0
4e21bb5794994a489ebd09ee30b6df70 | 127.4.5.66:39573 | HEALTHY | <none> | 3 | 0
5ad3a6abdbc846c1a6ed42269a100dd3 | 127.4.5.67:41365 | HEALTHY | <none> | 2 | 0
6f857824c6df4cb3a0eb69a4b66810f9 | 127.4.5.68:42071 | HEALTHY | <none> | 2 | 0
d305ba2a2a4640f7997613a7b8884dee | 127.4.5.65:39365 | HEALTHY | <none> | 3 | 0
e3fafd8e3e974740b2525f2e89d7cca5 | 127.4.5.69:35867 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 7
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 7 server(s) checked
local_ip_for_outbound_sockets | 127.4.5.65 | experimental | 127.4.5.65:39365
local_ip_for_outbound_sockets | 127.4.5.66 | experimental | 127.4.5.66:39573
local_ip_for_outbound_sockets | 127.4.5.67 | experimental | 127.4.5.67:41365
local_ip_for_outbound_sockets | 127.4.5.68 | experimental | 127.4.5.68:42071
local_ip_for_outbound_sockets | 127.4.5.69 | experimental | 127.4.5.69:35867
local_ip_for_outbound_sockets | 127.4.5.70 | experimental | 127.4.5.70:40279
local_ip_for_outbound_sockets | 127.4.5.71 | experimental | 127.4.5.71:37017
never_fsync | true | unsafe,advanced | all 7 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 7 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 7 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 7 server(s) checked
server_dump_info_format | pb | hidden | all 7 server(s) checked
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.4.5.65:39365
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.4.5.66:39573
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.4.5.67:41365
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.4.5.68:42071
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.4.5.69:35867
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.4.5.70:40279
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.4.5.71:37017
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+------------------+-------------------------
builtin_ntp_servers | 127.4.5.84:38437 | all 7 server(s) checked
time_source | builtin | all 7 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 8 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 3
Maximum | 3
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 7
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:12:03.198794 4117 log_verifier.cc:126] Checking tablet 0b7371b6f21b4786b16678e0798be3a3
I20240503 02:12:03.217504 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0b7371b6f21b4786b16678e0798be3a3
I20240503 02:12:03.217692 4117 log_verifier.cc:126] Checking tablet 0d9322f2472b4ef6a8e808b5c91948d3
I20240503 02:12:03.231513 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 0d9322f2472b4ef6a8e808b5c91948d3
I20240503 02:12:03.231652 4117 log_verifier.cc:126] Checking tablet 1c6cdade9a8241e4a0808deded2933f9
I20240503 02:12:03.246193 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1c6cdade9a8241e4a0808deded2933f9
I20240503 02:12:03.246337 4117 log_verifier.cc:126] Checking tablet 38eaf65d847745e49865a6cff720cd7e
I20240503 02:12:03.261163 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 38eaf65d847745e49865a6cff720cd7e
I20240503 02:12:03.261314 4117 log_verifier.cc:126] Checking tablet 587895f39d8b47858e704e8ccb274b89
I20240503 02:12:03.277033 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 587895f39d8b47858e704e8ccb274b89
I20240503 02:12:03.277166 4117 log_verifier.cc:126] Checking tablet 5c12714b4aab448e85708644e84e807c
I20240503 02:12:03.292605 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5c12714b4aab448e85708644e84e807c
I20240503 02:12:03.292755 4117 log_verifier.cc:126] Checking tablet 6caaa6a65dae42c994a6c451a34a41d0
I20240503 02:12:03.307044 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 6caaa6a65dae42c994a6c451a34a41d0
I20240503 02:12:03.307183 4117 log_verifier.cc:126] Checking tablet 83124ba50be244e3bcc43c41e9b644ec
I20240503 02:12:03.318874 4117 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 83124ba50be244e3bcc43c41e9b644ec
I20240503 02:12:03.319878 4117 log_verifier.cc:126] Checking tablet 840968df19714728a1c194dc65fa1b82
I20240503 02:12:03.337072 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 840968df19714728a1c194dc65fa1b82
I20240503 02:12:03.337322 4117 log_verifier.cc:126] Checking tablet 8a3067767dbf425c8a2a662ec01aa935
I20240503 02:12:03.354687 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8a3067767dbf425c8a2a662ec01aa935
I20240503 02:12:03.354835 4117 log_verifier.cc:126] Checking tablet 9bd51c9555b648d1baf4d1dcddc98fd6
I20240503 02:12:03.372931 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9bd51c9555b648d1baf4d1dcddc98fd6
I20240503 02:12:03.373076 4117 log_verifier.cc:126] Checking tablet b630de5974f3462697dc12edee49e592
I20240503 02:12:03.390837 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b630de5974f3462697dc12edee49e592
I20240503 02:12:03.390975 4117 log_verifier.cc:126] Checking tablet c041465d7d0f44f2af2beff9ed50ab96
I20240503 02:12:03.409054 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c041465d7d0f44f2af2beff9ed50ab96
I20240503 02:12:03.409216 4117 log_verifier.cc:126] Checking tablet d81a727ce55b4e629da976812205fb46
I20240503 02:12:03.426988 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d81a727ce55b4e629da976812205fb46
I20240503 02:12:03.427137 4117 log_verifier.cc:126] Checking tablet fe9f9a1a82914859bd08dd9f41de618a
I20240503 02:12:03.447466 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fe9f9a1a82914859bd08dd9f41de618a
I20240503 02:12:03.447631 4117 log_verifier.cc:126] Checking tablet ff6cd0b970c5434bac564d6c1b2b20e1
I20240503 02:12:03.466678 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ff6cd0b970c5434bac564d6c1b2b20e1
I20240503 02:12:03.470777 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.72:0
--local_ip_for_outbound_sockets=127.4.5.72
--webserver_interface=127.4.5.72
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--builtin_ntp_servers=127.4.5.84:38437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:12:03.913314 9356 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:03.913707 9356 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:03.913859 9356 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:03.922292 9356 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:03.922467 9356 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:03.922649 9356 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.72
I20240503 02:12:03.934062 9356 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:38437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.4.5.72
--webserver_port=0
--tserver_master_addrs=127.4.5.126:42813
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.72
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:03.935832 9356 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:03.938104 9356 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:03.954041 9364 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:03.954814 9361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:03.955283 9362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:03.956137 9356 server_base.cc:1034] running on GCE node
I20240503 02:12:03.956979 9356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:03.958250 9356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:03.959578 9356 hybrid_clock.cc:648] HybridClock initialized: now 1714702323959430 us; error 61 us; skew 500 ppm
I20240503 02:12:03.960043 9356 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:03.963238 9356 webserver.cc:460] Webserver started at http://127.4.5.72:40755/ using document root <none> and password file <none>
I20240503 02:12:03.964169 9356 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:03.964298 9356 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:03.964735 9356 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:03.968564 9356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "f5d946efbcdf4ec589950124b7890791"
format_stamp: "Formatted at 2024-05-03 02:12:03 on dist-test-slave-zv3f"
I20240503 02:12:03.969725 9356 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "f5d946efbcdf4ec589950124b7890791"
format_stamp: "Formatted at 2024-05-03 02:12:03 on dist-test-slave-zv3f"
I20240503 02:12:03.977608 9356 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20240503 02:12:03.982226 9370 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:03.984067 9356 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20240503 02:12:03.984325 9356 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "f5d946efbcdf4ec589950124b7890791"
format_stamp: "Formatted at 2024-05-03 02:12:03 on dist-test-slave-zv3f"
I20240503 02:12:03.984674 9356 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:04.013231 9356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:04.014374 9356 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:04.014721 9356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:04.016937 9356 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:04.018810 9356 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:04.018931 9356 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:04.019073 9356 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:04.019151 9356 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:04.065129 9356 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.72:44407
I20240503 02:12:04.065200 9483 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.72:44407 every 8 connection(s)
I20240503 02:12:04.066823 9356 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:04.068559 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 9356
I20240503 02:12:04.068945 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:12:04.100348 9484 heartbeater.cc:347] Connected to a master server at 127.4.5.126:42813
I20240503 02:12:04.100817 9484 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:04.102058 9484 heartbeater.cc:511] Master 127.4.5.126:42813 requested a full tablet report, sending...
I20240503 02:12:04.104183 6400 ts_manager.cc:194] Registered new tserver with Master: f5d946efbcdf4ec589950124b7890791 (127.4.5.72:44407)
I20240503 02:12:04.105325 6400 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.72:48459
I20240503 02:12:04.704759 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.733476 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.749938 6657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:04.754232 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.779946 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.785696 6526 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:04.820235 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:04.836994 9418 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:05.108791 9484 heartbeater.cc:503] Master 127.4.5.126:42813 was elected leader, sending a full tablet report...
I20240503 02:12:05.115551 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.116255 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.120321 6657 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:05.141829 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.161374 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.183362 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:05.183378 6526 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:05.225684 9418 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:05.307483 6546 consensus_queue.cc:237] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } }
I20240503 02:12:05.314247 9577 raft_consensus.cc:2932] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5d946efbcdf4ec589950124b7890791 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } } }
I20240503 02:12:05.320897 6401 catalog_manager.cc:5484] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee reported cstate change: config changed from index -1 to 4, NON_VOTER f5d946efbcdf4ec589950124b7890791 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:05.326352 6482 consensus_peers.cc:485] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee -> Peer f5d946efbcdf4ec589950124b7890791 (127.4.5.72:44407): Couldn't send request to peer f5d946efbcdf4ec589950124b7890791. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0d9322f2472b4ef6a8e808b5c91948d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:05.334415 6677 consensus_queue.cc:237] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } }
I20240503 02:12:05.336743 9589 raft_consensus.cc:2932] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f5d946efbcdf4ec589950124b7890791 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } } }
I20240503 02:12:05.342705 6401 catalog_manager.cc:5484] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 reported cstate change: config changed from index -1 to 4, NON_VOTER f5d946efbcdf4ec589950124b7890791 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:12:05.346526 6613 consensus_peers.cc:485] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 -> Peer f5d946efbcdf4ec589950124b7890791 (127.4.5.72:44407): Couldn't send request to peer f5d946efbcdf4ec589950124b7890791. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6caaa6a65dae42c994a6c451a34a41d0. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:05.826639 9623 ts_tablet_manager.cc:902] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: Initiating tablet copy from peer d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:12:05.828725 9623 tablet_copy_client.cc:308] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:39365
I20240503 02:12:05.837759 6566 tablet_copy_service.cc:140] P d305ba2a2a4640f7997613a7b8884dee: Received BeginTabletCopySession request for tablet 0d9322f2472b4ef6a8e808b5c91948d3 from peer f5d946efbcdf4ec589950124b7890791 ({username='slave'} at 127.4.5.72:54683)
I20240503 02:12:05.838003 6566 tablet_copy_service.cc:161] P d305ba2a2a4640f7997613a7b8884dee: Beginning new tablet copy session on tablet 0d9322f2472b4ef6a8e808b5c91948d3 from peer f5d946efbcdf4ec589950124b7890791 at {username='slave'} at 127.4.5.72:54683: session id = f5d946efbcdf4ec589950124b7890791-0d9322f2472b4ef6a8e808b5c91948d3
I20240503 02:12:05.841302 6566 tablet_copy_source_session.cc:205] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:05.847544 9623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0d9322f2472b4ef6a8e808b5c91948d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:05.859736 9623 tablet_copy_client.cc:783] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:05.860143 9623 tablet_copy_client.cc:647] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:05.862627 9623 tablet_copy_client.cc:523] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:05.866261 9623 tablet_bootstrap.cc:492] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: Bootstrap starting.
I20240503 02:12:05.877131 9623 log.cc:824] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:05.913203 9623 tablet_bootstrap.cc:492] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:05.913794 9623 tablet_bootstrap.cc:492] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: Bootstrap complete.
I20240503 02:12:05.914284 9623 ts_tablet_manager.cc:1367] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: Time spent bootstrapping tablet: real 0.048s user 0.043s sys 0.004s
I20240503 02:12:05.920213 9623 raft_consensus.cc:348] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } }
I20240503 02:12:05.920713 9623 raft_consensus.cc:729] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5d946efbcdf4ec589950124b7890791, State: Initialized, Role: LEARNER
I20240503 02:12:05.921252 9623 consensus_queue.cc:260] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } }
I20240503 02:12:05.923105 9623 ts_tablet_manager.cc:1398] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20240503 02:12:05.924336 6566 tablet_copy_service.cc:342] P d305ba2a2a4640f7997613a7b8884dee: Request end of tablet copy session f5d946efbcdf4ec589950124b7890791-0d9322f2472b4ef6a8e808b5c91948d3 received from {username='slave'} at 127.4.5.72:54683
I20240503 02:12:05.924523 6566 tablet_copy_service.cc:433] P d305ba2a2a4640f7997613a7b8884dee: ending tablet copy session f5d946efbcdf4ec589950124b7890791-0d9322f2472b4ef6a8e808b5c91948d3 on tablet 0d9322f2472b4ef6a8e808b5c91948d3 with peer f5d946efbcdf4ec589950124b7890791
I20240503 02:12:05.929554 9623 ts_tablet_manager.cc:902] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: Initiating tablet copy from peer 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:12:05.930590 9623 tablet_copy_client.cc:308] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:39573
I20240503 02:12:05.936286 6697 tablet_copy_service.cc:140] P 4e21bb5794994a489ebd09ee30b6df70: Received BeginTabletCopySession request for tablet 6caaa6a65dae42c994a6c451a34a41d0 from peer f5d946efbcdf4ec589950124b7890791 ({username='slave'} at 127.4.5.72:53979)
I20240503 02:12:05.936522 6697 tablet_copy_service.cc:161] P 4e21bb5794994a489ebd09ee30b6df70: Beginning new tablet copy session on tablet 6caaa6a65dae42c994a6c451a34a41d0 from peer f5d946efbcdf4ec589950124b7890791 at {username='slave'} at 127.4.5.72:53979: session id = f5d946efbcdf4ec589950124b7890791-6caaa6a65dae42c994a6c451a34a41d0
I20240503 02:12:05.938753 6697 tablet_copy_source_session.cc:205] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:05.940110 9623 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6caaa6a65dae42c994a6c451a34a41d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:05.944890 9623 tablet_copy_client.cc:783] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:05.945156 9623 tablet_copy_client.cc:647] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:05.947100 9623 tablet_copy_client.cc:523] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:05.949815 9623 tablet_bootstrap.cc:492] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: Bootstrap starting.
I20240503 02:12:05.981021 9623 tablet_bootstrap.cc:492] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:05.981534 9623 tablet_bootstrap.cc:492] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: Bootstrap complete.
I20240503 02:12:05.981850 9623 ts_tablet_manager.cc:1367] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: Time spent bootstrapping tablet: real 0.032s user 0.021s sys 0.010s
I20240503 02:12:05.982810 9623 raft_consensus.cc:348] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } }
I20240503 02:12:05.983039 9623 raft_consensus.cc:729] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f5d946efbcdf4ec589950124b7890791, State: Initialized, Role: LEARNER
I20240503 02:12:05.983247 9623 consensus_queue.cc:260] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: true } }
I20240503 02:12:05.983803 9623 ts_tablet_manager.cc:1398] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:05.984871 6697 tablet_copy_service.cc:342] P 4e21bb5794994a489ebd09ee30b6df70: Request end of tablet copy session f5d946efbcdf4ec589950124b7890791-6caaa6a65dae42c994a6c451a34a41d0 received from {username='slave'} at 127.4.5.72:53979
I20240503 02:12:05.985039 6697 tablet_copy_service.cc:433] P 4e21bb5794994a489ebd09ee30b6df70: ending tablet copy session f5d946efbcdf4ec589950124b7890791-6caaa6a65dae42c994a6c451a34a41d0 on tablet 6caaa6a65dae42c994a6c451a34a41d0 with peer f5d946efbcdf4ec589950124b7890791
I20240503 02:12:06.292912 9439 raft_consensus.cc:1203] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:12:06.543057 9439 raft_consensus.cc:1203] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:12:06.684367 9658 raft_consensus.cc:1050] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: attempting to promote NON_VOTER f5d946efbcdf4ec589950124b7890791 to VOTER
I20240503 02:12:06.685170 9658 consensus_queue.cc:237] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:06.687589 9439 raft_consensus.cc:1261] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 1 LEARNER]: Refusing update from remote peer d305ba2a2a4640f7997613a7b8884dee: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:06.688262 9658 consensus_queue.cc:1035] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:06.692610 9657 raft_consensus.cc:2932] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5d946efbcdf4ec589950124b7890791 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } } }
I20240503 02:12:06.693399 9439 raft_consensus.cc:2932] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5d946efbcdf4ec589950124b7890791 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } } }
I20240503 02:12:06.698745 6401 catalog_manager.cc:5484] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 reported cstate change: config changed from index 4 to 5, f5d946efbcdf4ec589950124b7890791 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "d305ba2a2a4640f7997613a7b8884dee" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } } }
I20240503 02:12:06.723567 6546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d9322f2472b4ef6a8e808b5c91948d3"
dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52654
I20240503 02:12:06.723817 6546 raft_consensus.cc:595] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:06.956151 6546 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "0d9322f2472b4ef6a8e808b5c91948d3"
dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:52676
I20240503 02:12:06.956475 6546 raft_consensus.cc:595] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:07.094059 9708 raft_consensus.cc:1050] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: attempting to promote NON_VOTER f5d946efbcdf4ec589950124b7890791 to VOTER
I20240503 02:12:07.094863 9708 consensus_queue.cc:237] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:07.097286 9439 raft_consensus.cc:1261] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 1 LEARNER]: Refusing update from remote peer 4e21bb5794994a489ebd09ee30b6df70: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:07.097916 9709 consensus_queue.cc:1035] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:07.101027 9708 raft_consensus.cc:2932] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5d946efbcdf4ec589950124b7890791 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } } }
I20240503 02:12:07.101706 9439 raft_consensus.cc:2932] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f5d946efbcdf4ec589950124b7890791 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } } }
I20240503 02:12:07.106789 6396 catalog_manager.cc:5484] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 reported cstate change: config changed from index 4 to 5, f5d946efbcdf4ec589950124b7890791 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "4e21bb5794994a489ebd09ee30b6df70" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } } }
I20240503 02:12:07.118367 9622 raft_consensus.cc:979] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: : Instructing follower f5d946efbcdf4ec589950124b7890791 to start an election
I20240503 02:12:07.118578 9622 raft_consensus.cc:1067] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Signalling peer f5d946efbcdf4ec589950124b7890791 to start an election
I20240503 02:12:07.119376 9439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "0d9322f2472b4ef6a8e808b5c91948d3"
dest_uuid: "f5d946efbcdf4ec589950124b7890791"
from {username='slave'} at 127.4.5.65:33621
I20240503 02:12:07.119746 9439 raft_consensus.cc:482] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:07.119972 9439 raft_consensus.cc:3037] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:07.123175 9439 raft_consensus.cc:504] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:07.124733 9439 leader_election.cc:288] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [CANDIDATE]: Term 2 election: Requested vote from peers d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365)
I20240503 02:12:07.125358 6546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0d9322f2472b4ef6a8e808b5c91948d3" candidate_uuid: "f5d946efbcdf4ec589950124b7890791" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "d305ba2a2a4640f7997613a7b8884dee"
I20240503 02:12:07.125615 6546 raft_consensus.cc:3032] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:07.125711 6546 raft_consensus.cc:729] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d305ba2a2a4640f7997613a7b8884dee, State: Running, Role: LEADER
I20240503 02:12:07.126014 6546 consensus_queue.cc:260] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:07.126322 6546 raft_consensus.cc:3037] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:07.128435 6546 raft_consensus.cc:2454] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d946efbcdf4ec589950124b7890791 in term 2.
I20240503 02:12:07.129074 9372 leader_election.cc:302] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: d305ba2a2a4640f7997613a7b8884dee, f5d946efbcdf4ec589950124b7890791; no voters:
I20240503 02:12:07.129498 9630 raft_consensus.cc:2781] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:07.129992 9630 raft_consensus.cc:686] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 2 LEADER]: Becoming Leader. State: Replica: f5d946efbcdf4ec589950124b7890791, State: Running, Role: LEADER
I20240503 02:12:07.130558 9630 consensus_queue.cc:237] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:07.135824 6401 catalog_manager.cc:5484] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 reported cstate change: term changed from 1 to 2, leader changed from d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) to f5d946efbcdf4ec589950124b7890791 (127.4.5.72). New cstate: current_term: 2 leader_uuid: "f5d946efbcdf4ec589950124b7890791" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:12:07.147408 6391 catalog_manager.cc:5014] ChangeConfig:REMOVE_PEER RPC for tablet 0d9322f2472b4ef6a8e808b5c91948d3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER failed with leader f5d946efbcdf4ec589950124b7890791 (127.4.5.72:44407) due to error UNKNOWN_ERROR; will retry: Illegal state: Leader has not yet committed an operation in its own term
I20240503 02:12:07.177496 6677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6caaa6a65dae42c994a6c451a34a41d0"
dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39166
I20240503 02:12:07.177793 6677 raft_consensus.cc:595] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:07.409759 6677 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "6caaa6a65dae42c994a6c451a34a41d0"
dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:39186
I20240503 02:12:07.410087 6677 raft_consensus.cc:595] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:07.513171 9709 raft_consensus.cc:979] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: : Instructing follower f5d946efbcdf4ec589950124b7890791 to start an election
I20240503 02:12:07.513409 9709 raft_consensus.cc:1067] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Signalling peer f5d946efbcdf4ec589950124b7890791 to start an election
I20240503 02:12:07.514065 9439 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "6caaa6a65dae42c994a6c451a34a41d0"
dest_uuid: "f5d946efbcdf4ec589950124b7890791"
from {username='slave'} at 127.4.5.66:42149
I20240503 02:12:07.514302 9439 raft_consensus.cc:482] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:07.514400 9439 raft_consensus.cc:3037] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:07.516528 9439 raft_consensus.cc:504] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:07.517145 9439 leader_election.cc:288] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [CANDIDATE]: Term 2 election: Requested vote from peers 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573)
I20240503 02:12:07.517808 6677 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6caaa6a65dae42c994a6c451a34a41d0" candidate_uuid: "f5d946efbcdf4ec589950124b7890791" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "4e21bb5794994a489ebd09ee30b6df70"
I20240503 02:12:07.518043 6677 raft_consensus.cc:3032] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:07.518129 6677 raft_consensus.cc:729] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4e21bb5794994a489ebd09ee30b6df70, State: Running, Role: LEADER
I20240503 02:12:07.518431 6677 consensus_queue.cc:260] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:07.518713 6677 raft_consensus.cc:3037] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:07.521013 6677 raft_consensus.cc:2454] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f5d946efbcdf4ec589950124b7890791 in term 2.
I20240503 02:12:07.521485 9371 leader_election.cc:302] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 4e21bb5794994a489ebd09ee30b6df70, f5d946efbcdf4ec589950124b7890791; no voters:
I20240503 02:12:07.521807 9630 raft_consensus.cc:2781] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:07.521981 9630 raft_consensus.cc:686] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 2 LEADER]: Becoming Leader. State: Replica: f5d946efbcdf4ec589950124b7890791, State: Running, Role: LEADER
I20240503 02:12:07.522277 9630 consensus_queue.cc:237] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:07.524819 6401 catalog_manager.cc:5484] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 reported cstate change: term changed from 1 to 2, leader changed from 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) to f5d946efbcdf4ec589950124b7890791 (127.4.5.72). New cstate: current_term: 2 leader_uuid: "f5d946efbcdf4ec589950124b7890791" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:07.597612 6546 raft_consensus.cc:1261] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Refusing update from remote peer f5d946efbcdf4ec589950124b7890791: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:07.598443 9630 consensus_queue.cc:1035] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d305ba2a2a4640f7997613a7b8884dee" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 39365 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:07.774391 9439 consensus_queue.cc:237] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:07.775609 9749 raft_consensus.cc:2932] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } } }
I20240503 02:12:07.779996 6391 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0d9322f2472b4ef6a8e808b5c91948d3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:07.781535 6401 catalog_manager.cc:5484] T 0d9322f2472b4ef6a8e808b5c91948d3 P f5d946efbcdf4ec589950124b7890791 reported cstate change: config changed from index 5 to 7, VOTER d305ba2a2a4640f7997613a7b8884dee (127.4.5.65) evicted. New cstate: current_term: 2 leader_uuid: "f5d946efbcdf4ec589950124b7890791" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:07.785270 6526 tablet_service.cc:1514] Processing DeleteTablet for tablet 0d9322f2472b4ef6a8e808b5c91948d3 with delete_type TABLET_DATA_TOMBSTONED (TS d305ba2a2a4640f7997613a7b8884dee not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:50604
I20240503 02:12:07.785949 9760 tablet_replica.cc:330] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: stopping tablet replica
I20240503 02:12:07.786159 9760 raft_consensus.cc:2229] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:07.786381 9760 raft_consensus.cc:2258] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:07.788019 9760 ts_tablet_manager.cc:1875] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:07.793030 9760 ts_tablet_manager.cc:1888] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:07.793175 9760 log.cc:1196] T 0d9322f2472b4ef6a8e808b5c91948d3 P d305ba2a2a4640f7997613a7b8884dee: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/0d9322f2472b4ef6a8e808b5c91948d3
I20240503 02:12:07.794127 6389 catalog_manager.cc:4830] TS d305ba2a2a4640f7997613a7b8884dee (127.4.5.65:39365): tablet 0d9322f2472b4ef6a8e808b5c91948d3 (table rebalance_test_table_2 [id=efe14e9c57a44f339179a831adbcd921]) successfully deleted
I20240503 02:12:08.127213 6677 raft_consensus.cc:1261] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Refusing update from remote peer f5d946efbcdf4ec589950124b7890791: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:08.127913 9749 consensus_queue.cc:1035] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4e21bb5794994a489ebd09ee30b6df70" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 39573 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:08.162134 9439 consensus_queue.cc:237] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } }
I20240503 02:12:08.163187 9749 raft_consensus.cc:2932] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } } }
I20240503 02:12:08.166395 6391 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 6caaa6a65dae42c994a6c451a34a41d0 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 6)
I20240503 02:12:08.167727 6396 catalog_manager.cc:5484] T 6caaa6a65dae42c994a6c451a34a41d0 P f5d946efbcdf4ec589950124b7890791 reported cstate change: config changed from index 5 to 7, VOTER 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66) evicted. New cstate: current_term: 2 leader_uuid: "f5d946efbcdf4ec589950124b7890791" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "f5d946efbcdf4ec589950124b7890791" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 44407 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:08.171687 6657 tablet_service.cc:1514] Processing DeleteTablet for tablet 6caaa6a65dae42c994a6c451a34a41d0 with delete_type TABLET_DATA_TOMBSTONED (TS 4e21bb5794994a489ebd09ee30b6df70 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:35614
I20240503 02:12:08.172490 9785 tablet_replica.cc:330] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: stopping tablet replica
I20240503 02:12:08.172703 9785 raft_consensus.cc:2229] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:08.172926 9785 raft_consensus.cc:2258] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:08.174582 9785 ts_tablet_manager.cc:1875] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:08.179870 9785 ts_tablet_manager.cc:1888] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:08.180016 9785 log.cc:1196] T 6caaa6a65dae42c994a6c451a34a41d0 P 4e21bb5794994a489ebd09ee30b6df70: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/6caaa6a65dae42c994a6c451a34a41d0
I20240503 02:12:08.181025 6388 catalog_manager.cc:4830] TS 4e21bb5794994a489ebd09ee30b6df70 (127.4.5.66:39573): tablet 6caaa6a65dae42c994a6c451a34a41d0 (table rebalance_test_table_3 [id=107814137f694dceb7cfcaae16c5fd8f]) successfully deleted
I20240503 02:12:08.418018 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.424299 6657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.456094 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.468947 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.475373 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.483423 9418 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.483501 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.504809 6526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.705190 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.712904 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.713060 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.716722 6657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.754841 6526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.759744 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.770972 9418 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:08.772536 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.095094 8197 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.095885 8331 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.126394 7783 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.131995 6657 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.147099 7920 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.172341 6526 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.180035 8064 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:09.204396 9418 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
Master Summary
UUID | Address | Status
----------------------------------+-------------------+---------
596ca00a45524913952aa8c8b8864eee | 127.4.5.126:42813 | HEALTHY
Unusual flags for Master:
Flag | Value | Tags | Master
------------------------------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
allow_unsafe_replication_factor | true | runtime,unsafe | all 1 server(s) checked
ipki_ca_key_size | 768 | experimental | all 1 server(s) checked
ipki_server_key_size | 768 | experimental | all 1 server(s) checked
never_fsync | true | unsafe,advanced | all 1 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 1 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 1 server(s) checked
rpc_reuseport | true | experimental | all 1 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 1 server(s) checked
server_dump_info_format | pb | hidden | all 1 server(s) checked
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb | hidden | all 1 server(s) checked
tsk_num_rsa_bits | 512 | experimental | all 1 server(s) checked
Flags of checked categories for Master:
Flag | Value | Master
---------------------+------------------+-------------------------
builtin_ntp_servers | 127.4.5.84:38437 | all 1 server(s) checked
time_source | builtin | all 1 server(s) checked
Tablet Server Summary
UUID | Address | Status | Location | Tablet Leaders | Active Scanners
----------------------------------+------------------+---------+----------+----------------+-----------------
329369f9124942438b3176e3b0bacf98 | 127.4.5.70:40279 | HEALTHY | <none> | 2 | 0
4bf9c9a4cde949dbabf5144188edbf5b | 127.4.5.71:37017 | HEALTHY | <none> | 2 | 0
4e21bb5794994a489ebd09ee30b6df70 | 127.4.5.66:39573 | HEALTHY | <none> | 2 | 0
5ad3a6abdbc846c1a6ed42269a100dd3 | 127.4.5.67:41365 | HEALTHY | <none> | 2 | 0
6f857824c6df4cb3a0eb69a4b66810f9 | 127.4.5.68:42071 | HEALTHY | <none> | 2 | 0
d305ba2a2a4640f7997613a7b8884dee | 127.4.5.65:39365 | HEALTHY | <none> | 2 | 0
e3fafd8e3e974740b2525f2e89d7cca5 | 127.4.5.69:35867 | HEALTHY | <none> | 2 | 0
f5d946efbcdf4ec589950124b7890791 | 127.4.5.72:44407 | HEALTHY | <none> | 2 | 0
Tablet Server Location Summary
Location | Count
----------+---------
<none> | 8
Unusual flags for Tablet Server:
Flag | Value | Tags | Tablet Server
------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------+-------------------------
ipki_server_key_size | 768 | experimental | all 8 server(s) checked
local_ip_for_outbound_sockets | 127.4.5.65 | experimental | 127.4.5.65:39365
local_ip_for_outbound_sockets | 127.4.5.66 | experimental | 127.4.5.66:39573
local_ip_for_outbound_sockets | 127.4.5.67 | experimental | 127.4.5.67:41365
local_ip_for_outbound_sockets | 127.4.5.68 | experimental | 127.4.5.68:42071
local_ip_for_outbound_sockets | 127.4.5.69 | experimental | 127.4.5.69:35867
local_ip_for_outbound_sockets | 127.4.5.70 | experimental | 127.4.5.70:40279
local_ip_for_outbound_sockets | 127.4.5.71 | experimental | 127.4.5.71:37017
local_ip_for_outbound_sockets | 127.4.5.72 | experimental | 127.4.5.72:44407
never_fsync | true | unsafe,advanced | all 8 server(s) checked
openssl_security_level_override | 0 | unsafe,hidden | all 8 server(s) checked
raft_prepare_replacement_before_eviction | true | experimental,advanced | all 8 server(s) checked
rpc_server_allow_ephemeral_ports | true | unsafe | all 8 server(s) checked
server_dump_info_format | pb | hidden | all 8 server(s) checked
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb | hidden | 127.4.5.65:39365
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb | hidden | 127.4.5.66:39573
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb | hidden | 127.4.5.67:41365
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb | hidden | 127.4.5.68:42071
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb | hidden | 127.4.5.69:35867
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb | hidden | 127.4.5.70:40279
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb | hidden | 127.4.5.71:37017
server_dump_info_path | /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.RebalanceParamTest.Rebalance_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb | hidden | 127.4.5.72:44407
Flags of checked categories for Tablet Server:
Flag | Value | Tablet Server
---------------------+------------------+-------------------------
builtin_ntp_servers | 127.4.5.84:38437 | all 8 server(s) checked
time_source | builtin | all 8 server(s) checked
Version Summary
Version | Servers
-----------------+-------------------------
1.18.0-SNAPSHOT | all 9 server(s) checked
Tablet Summary
The cluster doesn't have any matching system tables
Summary by table
Name | RF | Status | Total Tablets | Healthy | Recovering | Under-replicated | Unavailable
------------------------+----+---------+---------------+---------+------------+------------------+-------------
TestTable | 1 | HEALTHY | 1 | 1 | 0 | 0 | 0
rebalance_test_table_0 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_1 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_2 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_3 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
rebalance_test_table_4 | 1 | HEALTHY | 3 | 3 | 0 | 0 | 0
Tablet Replica Count Summary
Statistic | Replica Count
----------------+---------------
Minimum | 2
First Quartile | 2
Median | 2
Third Quartile | 2
Maximum | 2
Total Count Summary
| Total Count
----------------+-------------
Masters | 1
Tablet Servers | 8
Tables | 6
Tablets | 16
Replicas | 16
==================
Warnings:
==================
Some masters have unsafe, experimental, or hidden flags set
Some tablet servers have unsafe, experimental, or hidden flags set
OK
I20240503 02:12:09.252996 4117 log_verifier.cc:126] Checking tablet 0b7371b6f21b4786b16678e0798be3a3
I20240503 02:12:09.264425 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0b7371b6f21b4786b16678e0798be3a3
I20240503 02:12:09.264631 4117 log_verifier.cc:126] Checking tablet 0d9322f2472b4ef6a8e808b5c91948d3
I20240503 02:12:09.279843 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 0d9322f2472b4ef6a8e808b5c91948d3
I20240503 02:12:09.279979 4117 log_verifier.cc:126] Checking tablet 1c6cdade9a8241e4a0808deded2933f9
I20240503 02:12:09.291520 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 1c6cdade9a8241e4a0808deded2933f9
I20240503 02:12:09.291642 4117 log_verifier.cc:126] Checking tablet 38eaf65d847745e49865a6cff720cd7e
I20240503 02:12:09.303120 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 38eaf65d847745e49865a6cff720cd7e
I20240503 02:12:09.303249 4117 log_verifier.cc:126] Checking tablet 587895f39d8b47858e704e8ccb274b89
I20240503 02:12:09.314522 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 587895f39d8b47858e704e8ccb274b89
I20240503 02:12:09.314637 4117 log_verifier.cc:126] Checking tablet 5c12714b4aab448e85708644e84e807c
I20240503 02:12:09.326037 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 5c12714b4aab448e85708644e84e807c
I20240503 02:12:09.326167 4117 log_verifier.cc:126] Checking tablet 6caaa6a65dae42c994a6c451a34a41d0
I20240503 02:12:09.340638 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 6caaa6a65dae42c994a6c451a34a41d0
I20240503 02:12:09.340776 4117 log_verifier.cc:126] Checking tablet 83124ba50be244e3bcc43c41e9b644ec
I20240503 02:12:09.346772 4117 log_verifier.cc:177] Verified matching terms for 2 ops in tablet 83124ba50be244e3bcc43c41e9b644ec
I20240503 02:12:09.346884 4117 log_verifier.cc:126] Checking tablet 840968df19714728a1c194dc65fa1b82
I20240503 02:12:09.356876 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 840968df19714728a1c194dc65fa1b82
I20240503 02:12:09.356989 4117 log_verifier.cc:126] Checking tablet 8a3067767dbf425c8a2a662ec01aa935
I20240503 02:12:09.367033 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet 8a3067767dbf425c8a2a662ec01aa935
I20240503 02:12:09.367151 4117 log_verifier.cc:126] Checking tablet 9bd51c9555b648d1baf4d1dcddc98fd6
I20240503 02:12:09.378636 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet 9bd51c9555b648d1baf4d1dcddc98fd6
I20240503 02:12:09.378772 4117 log_verifier.cc:126] Checking tablet b630de5974f3462697dc12edee49e592
I20240503 02:12:09.389732 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet b630de5974f3462697dc12edee49e592
I20240503 02:12:09.389858 4117 log_verifier.cc:126] Checking tablet c041465d7d0f44f2af2beff9ed50ab96
I20240503 02:12:09.400738 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet c041465d7d0f44f2af2beff9ed50ab96
I20240503 02:12:09.400851 4117 log_verifier.cc:126] Checking tablet d81a727ce55b4e629da976812205fb46
I20240503 02:12:09.411027 4117 log_verifier.cc:177] Verified matching terms for 3 ops in tablet d81a727ce55b4e629da976812205fb46
I20240503 02:12:09.411154 4117 log_verifier.cc:126] Checking tablet fe9f9a1a82914859bd08dd9f41de618a
I20240503 02:12:09.422205 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet fe9f9a1a82914859bd08dd9f41de618a
I20240503 02:12:09.422333 4117 log_verifier.cc:126] Checking tablet ff6cd0b970c5434bac564d6c1b2b20e1
I20240503 02:12:09.432972 4117 log_verifier.cc:177] Verified matching terms for 7 ops in tablet ff6cd0b970c5434bac564d6c1b2b20e1
I20240503 02:12:09.434077 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 6464
I20240503 02:12:09.435720 6586 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:09.600884 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 6464
I20240503 02:12:09.628515 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 6595
I20240503 02:12:09.629715 6716 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:09.788723 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 6595
I20240503 02:12:09.815989 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 7720
I20240503 02:12:09.822930 7845 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:09.955919 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 7720
I20240503 02:12:09.979638 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 7854
I20240503 02:12:09.986837 7982 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:10.121555 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 7854
I20240503 02:12:10.144731 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 7996
I20240503 02:12:10.152354 8124 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:10.284391 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 7996
I20240503 02:12:10.306067 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 8133
I20240503 02:12:10.314751 8258 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:10.450527 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 8133
I20240503 02:12:10.475804 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 8268
I20240503 02:12:10.482414 8391 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:10.612875 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 8268
I20240503 02:12:10.634924 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 9356
I20240503 02:12:10.641798 9478 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:10.755667 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 9356
I20240503 02:12:10.776499 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 6373
I20240503 02:12:10.777813 6434 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:10.956388 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 6373
2024-05-03T02:12:10Z chronyd exiting
[ OK ] RebalanceParamTest.Rebalance/1 (30038 ms)
[----------] 1 test from RebalanceParamTest (30038 ms total)
[----------] 1 test from DDLDuringRebalancingTest
[ RUN ] DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing/1
I20240503 02:12:11.028066 4117 test_util.cc:268] Using random seed: 1062609476
I20240503 02:12:11.030725 4117 ts_itest-base.cc:112] Starting cluster with:
I20240503 02:12:11.030800 4117 ts_itest-base.cc:113] --------------
I20240503 02:12:11.030846 4117 ts_itest-base.cc:114] 8 tablet servers
I20240503 02:12:11.030885 4117 ts_itest-base.cc:115] 3 replicas per TS
I20240503 02:12:11.030944 4117 ts_itest-base.cc:116] --------------
2024-05-03T02:12:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2024-05-03T02:12:11Z Disabled control of system clock
I20240503 02:12:11.064919 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.5.126:35467
--webserver_interface=127.4.5.126
--webserver_port=0
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.4.5.126:35467
--tserver_unresponsive_timeout_ms=3000
--enable_range_replica_placement=false
--raft_prepare_replacement_before_eviction=true
--enable_range_replica_placement=false with env {}
W20240503 02:12:11.457175 9921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:11.457613 9921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:11.457727 9921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:11.466691 9921 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:11.466866 9921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20240503 02:12:11.466928 9921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:11.466989 9921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20240503 02:12:11.467039 9921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20240503 02:12:11.477324 9921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
--enable_range_replica_placement=false
--ipki_ca_key_size=768
--master_addresses=127.4.5.126:35467
--tserver_unresponsive_timeout_ms=3000
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.4.5.126:35467
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
--webserver_interface=127.4.5.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:11.479044 9921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:11.481274 9921 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:11.494554 9927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:11.495087 9929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:11.495607 9926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:11.497074 9921 server_base.cc:1034] running on GCE node
I20240503 02:12:11.498072 9921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:11.499812 9921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:11.501017 9921 hybrid_clock.cc:648] HybridClock initialized: now 1714702331500963 us; error 48 us; skew 500 ppm
I20240503 02:12:11.501415 9921 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:11.511116 9921 webserver.cc:460] Webserver started at http://127.4.5.126:44171/ using document root <none> and password file <none>
I20240503 02:12:11.512042 9921 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:11.512172 9921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:11.512595 9921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:11.515770 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/instance:
uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9"
format_stamp: "Formatted at 2024-05-03 02:12:11 on dist-test-slave-zv3f"
I20240503 02:12:11.516788 9921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal/instance:
uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9"
format_stamp: "Formatted at 2024-05-03 02:12:11 on dist-test-slave-zv3f"
I20240503 02:12:11.522842 9921 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20240503 02:12:11.526635 9935 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:11.528115 9921 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:11.528307 9921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9"
format_stamp: "Formatted at 2024-05-03 02:12:11 on dist-test-slave-zv3f"
I20240503 02:12:11.528550 9921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:11.542053 9921 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:11.543043 9921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:11.543347 9921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:11.563946 9921 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.126:35467
I20240503 02:12:11.563966 9986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.126:35467 every 8 connection(s)
I20240503 02:12:11.565603 9921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/data/info.pb
I20240503 02:12:11.569552 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 9921
I20240503 02:12:11.569769 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/master-0/wal/instance
I20240503 02:12:11.570017 9987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:11.584602 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Bootstrap starting.
I20240503 02:12:11.588150 9987 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:11.589330 9987 log.cc:824] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:11.592053 9987 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: No bootstrap required, opened a new log
I20240503 02:12:11.598423 9987 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35467 } }
I20240503 02:12:11.598774 9987 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:11.598855 9987 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1b81c1c3148f4f3eabfcd10eecb5c6b9, State: Initialized, Role: FOLLOWER
I20240503 02:12:11.599431 9987 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35467 } }
I20240503 02:12:11.599613 9987 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20240503 02:12:11.599712 9987 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20240503 02:12:11.599829 9987 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:11.601994 9987 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35467 } }
I20240503 02:12:11.602463 9987 leader_election.cc:302] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1b81c1c3148f4f3eabfcd10eecb5c6b9; no voters:
I20240503 02:12:11.603051 9987 leader_election.cc:288] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20240503 02:12:11.603199 9992 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:11.603915 9992 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [term 1 LEADER]: Becoming Leader. State: Replica: 1b81c1c3148f4f3eabfcd10eecb5c6b9, State: Running, Role: LEADER
I20240503 02:12:11.604562 9992 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35467 } }
I20240503 02:12:11.605342 9987 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [sys.catalog]: configured and running, proceeding with master startup.
I20240503 02:12:11.607549 9994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1b81c1c3148f4f3eabfcd10eecb5c6b9. Latest consensus state: current_term: 1 leader_uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35467 } } }
I20240503 02:12:11.607575 9993 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1b81c1c3148f4f3eabfcd10eecb5c6b9" member_type: VOTER last_known_addr { host: "127.4.5.126" port: 35467 } } }
I20240503 02:12:11.607928 9994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:11.608008 9993 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9 [sys.catalog]: This master's current role is: LEADER
I20240503 02:12:11.610239 9997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20240503 02:12:11.615754 9997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20240503 02:12:11.624961 9997 catalog_manager.cc:1347] Generated new cluster ID: aebefcaa4d0e4b18a0a3057117c72a4a
I20240503 02:12:11.625099 9997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20240503 02:12:11.652202 9997 catalog_manager.cc:1370] Generated new certificate authority record
I20240503 02:12:11.653702 9997 catalog_manager.cc:1504] Loading token signing keys...
I20240503 02:12:11.665513 9997 catalog_manager.cc:5857] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Generated new TSK 0
I20240503 02:12:11.666587 9997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20240503 02:12:11.686482 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.65:0
--local_ip_for_outbound_sockets=127.4.5.65
--webserver_interface=127.4.5.65
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:12:12.069388 10011 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:12.069803 10011 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:12.069921 10011 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:12.078157 10011 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:12.078331 10011 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:12.078466 10011 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.65
I20240503 02:12:12.088627 10011 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
--webserver_interface=127.4.5.65
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.65
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:12.090298 10011 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:12.092378 10011 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:12.106387 10019 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.107026 10016 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.107712 10017 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:12.108556 10011 server_base.cc:1034] running on GCE node
I20240503 02:12:12.109282 10011 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:12.110424 10011 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:12.111634 10011 hybrid_clock.cc:648] HybridClock initialized: now 1714702332111584 us; error 48 us; skew 500 ppm
I20240503 02:12:12.112030 10011 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:12.114744 10011 webserver.cc:460] Webserver started at http://127.4.5.65:33073/ using document root <none> and password file <none>
I20240503 02:12:12.115636 10011 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:12.115767 10011 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:12.116128 10011 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:12.119163 10011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/instance:
uuid: "031a0afb01b6469299fa829fa3467ef7"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-zv3f"
I20240503 02:12:12.120002 10011 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/instance:
uuid: "031a0afb01b6469299fa829fa3467ef7"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-zv3f"
I20240503 02:12:12.125725 10011 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20240503 02:12:12.129454 10025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:12.130973 10011 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:12:12.131132 10011 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
uuid: "031a0afb01b6469299fa829fa3467ef7"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-zv3f"
I20240503 02:12:12.131351 10011 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:12.156814 10011 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:12.158043 10011 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:12.158375 10011 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:12.159503 10011 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:12.161360 10011 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:12.161473 10011 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:12.161602 10011 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:12.161697 10011 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:12.197975 10011 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.65:38119
I20240503 02:12:12.198055 10139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.65:38119 every 8 connection(s)
I20240503 02:12:12.199733 10011 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/data/info.pb
I20240503 02:12:12.202396 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 10011
I20240503 02:12:12.202633 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/instance
I20240503 02:12:12.206892 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.66:0
--local_ip_for_outbound_sockets=127.4.5.66
--webserver_interface=127.4.5.66
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:12.215059 10140 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:12.215416 10140 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:12.216392 10140 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:12.218791 9952 ts_manager.cc:194] Registered new tserver with Master: 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:12.220533 9952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.65:45131
W20240503 02:12:12.586061 10144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:12.586417 10144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:12.586561 10144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:12.594929 10144 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:12.595088 10144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:12.595280 10144 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.66
I20240503 02:12:12.604863 10144 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
--webserver_interface=127.4.5.66
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.66
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:12.606585 10144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:12.608700 10144 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:12.622200 10149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.622534 10152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:12.623215 10150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:12.623863 10144 server_base.cc:1034] running on GCE node
I20240503 02:12:12.624547 10144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:12.625597 10144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:12.626806 10144 hybrid_clock.cc:648] HybridClock initialized: now 1714702332626759 us; error 45 us; skew 500 ppm
I20240503 02:12:12.627131 10144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:12.629712 10144 webserver.cc:460] Webserver started at http://127.4.5.66:41979/ using document root <none> and password file <none>
I20240503 02:12:12.630513 10144 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:12.630628 10144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:12.630956 10144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:12.634017 10144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/instance:
uuid: "eff8e8d11a4447218741bb994aa41082"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-zv3f"
I20240503 02:12:12.634878 10144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/instance:
uuid: "eff8e8d11a4447218741bb994aa41082"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-zv3f"
I20240503 02:12:12.641253 10144 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.002s
I20240503 02:12:12.644907 10158 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:12.646348 10144 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:12:12.646514 10144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
uuid: "eff8e8d11a4447218741bb994aa41082"
format_stamp: "Formatted at 2024-05-03 02:12:12 on dist-test-slave-zv3f"
I20240503 02:12:12.646733 10144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:12.686056 10144 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:12.687165 10144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:12.687472 10144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:12.688611 10144 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:12.690336 10144 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:12.690464 10144 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:12.690582 10144 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:12.690665 10144 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:12.726269 10144 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.66:42397
I20240503 02:12:12.726330 10272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.66:42397 every 8 connection(s)
I20240503 02:12:12.727855 10144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/data/info.pb
I20240503 02:12:12.733582 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 10144
I20240503 02:12:12.733804 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/instance
I20240503 02:12:12.737663 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.67:0
--local_ip_for_outbound_sockets=127.4.5.67
--webserver_interface=127.4.5.67
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:12.742861 10273 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:12.743175 10273 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:12.744071 10273 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:12.745852 9952 ts_manager.cc:194] Registered new tserver with Master: eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:12.746673 9952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.66:42979
W20240503 02:12:13.124347 10277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:13.124851 10277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:13.125027 10277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:13.133199 10277 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:13.133323 10277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:13.133458 10277 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.67
I20240503 02:12:13.143337 10277 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
--webserver_interface=127.4.5.67
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.67
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:13.145028 10277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:13.147028 10277 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:13.160961 10283 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:13.161535 10285 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:13.162134 10282 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:13.162904 10277 server_base.cc:1034] running on GCE node
I20240503 02:12:13.163583 10277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:13.164712 10277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:13.165915 10277 hybrid_clock.cc:648] HybridClock initialized: now 1714702333165857 us; error 49 us; skew 500 ppm
I20240503 02:12:13.166270 10277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:13.168839 10277 webserver.cc:460] Webserver started at http://127.4.5.67:33169/ using document root <none> and password file <none>
I20240503 02:12:13.169695 10277 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:13.169818 10277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:13.170163 10277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:13.173336 10277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/instance:
uuid: "47f901e617024baaa1865d12261ef84f"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-zv3f"
I20240503 02:12:13.174213 10277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/instance:
uuid: "47f901e617024baaa1865d12261ef84f"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-zv3f"
I20240503 02:12:13.179968 10277 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20240503 02:12:13.183588 10291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.185086 10277 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:13.185257 10277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
uuid: "47f901e617024baaa1865d12261ef84f"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-zv3f"
I20240503 02:12:13.185472 10277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:13.209010 10277 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:13.209968 10277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:13.210285 10277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:13.211361 10277 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:13.213104 10277 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:13.213217 10277 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.213333 10277 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:13.213407 10277 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.223836 10140 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
I20240503 02:12:13.252334 10277 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.67:41899
I20240503 02:12:13.252369 10404 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.67:41899 every 8 connection(s)
I20240503 02:12:13.253940 10277 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/data/info.pb
I20240503 02:12:13.255365 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 10277
I20240503 02:12:13.255606 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/instance
I20240503 02:12:13.259831 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.68:0
--local_ip_for_outbound_sockets=127.4.5.68
--webserver_interface=127.4.5.68
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:13.269205 10405 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:13.269613 10405 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:13.270900 10405 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:13.272776 9952 ts_manager.cc:194] Registered new tserver with Master: 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:13.273684 9952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.67:52855
W20240503 02:12:13.648896 10409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:13.649313 10409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:13.649472 10409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:13.657582 10409 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:13.657769 10409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:13.657958 10409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.68
I20240503 02:12:13.667799 10409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.68:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
--webserver_interface=127.4.5.68
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.68
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:13.669641 10409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:13.671690 10409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:13.685462 10415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:13.686111 10417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:13.686781 10414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:13.687515 10409 server_base.cc:1034] running on GCE node
I20240503 02:12:13.688205 10409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:13.689314 10409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:13.690522 10409 hybrid_clock.cc:648] HybridClock initialized: now 1714702333690461 us; error 58 us; skew 500 ppm
I20240503 02:12:13.690915 10409 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:13.693490 10409 webserver.cc:460] Webserver started at http://127.4.5.68:44881/ using document root <none> and password file <none>
I20240503 02:12:13.694304 10409 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:13.694414 10409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:13.694758 10409 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:13.697824 10409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/instance:
uuid: "c6ec2f1283994a3089741715f811c850"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-zv3f"
I20240503 02:12:13.698644 10409 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/instance:
uuid: "c6ec2f1283994a3089741715f811c850"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-zv3f"
I20240503 02:12:13.704190 10409 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 02:12:13.707963 10423 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.709558 10409 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20240503 02:12:13.709725 10409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
uuid: "c6ec2f1283994a3089741715f811c850"
format_stamp: "Formatted at 2024-05-03 02:12:13 on dist-test-slave-zv3f"
I20240503 02:12:13.709924 10409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:13.739328 10409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:13.740327 10409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:13.740648 10409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:13.741732 10409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:13.743412 10409 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:13.743503 10409 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.743593 10409 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:13.743651 10409 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:13.749390 10273 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
I20240503 02:12:13.778309 10409 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.68:35025
I20240503 02:12:13.778364 10536 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.68:35025 every 8 connection(s)
I20240503 02:12:13.779862 10409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/data/info.pb
I20240503 02:12:13.788560 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 10409
I20240503 02:12:13.788758 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/instance
I20240503 02:12:13.792361 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.69:0
--local_ip_for_outbound_sockets=127.4.5.69
--webserver_interface=127.4.5.69
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:13.794939 10537 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:13.795279 10537 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:13.796272 10537 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:13.798045 9952 ts_manager.cc:194] Registered new tserver with Master: c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:13.799290 9952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.68:36773
W20240503 02:12:14.173812 10541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:14.174192 10541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:14.174348 10541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:14.182632 10541 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:14.182827 10541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:14.183017 10541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.69
I20240503 02:12:14.193061 10541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.69:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.4.5.69
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.69
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:14.194744 10541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:14.197101 10541 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:14.210909 10546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:14.211439 10549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:14.212080 10547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:14.212836 10541 server_base.cc:1034] running on GCE node
I20240503 02:12:14.213423 10541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:14.214404 10541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:14.215621 10541 hybrid_clock.cc:648] HybridClock initialized: now 1714702334215566 us; error 41 us; skew 500 ppm
I20240503 02:12:14.215953 10541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:14.218418 10541 webserver.cc:460] Webserver started at http://127.4.5.69:38057/ using document root <none> and password file <none>
I20240503 02:12:14.219237 10541 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:14.219352 10541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:14.219686 10541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:14.222540 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/instance:
uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-zv3f"
I20240503 02:12:14.223364 10541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/instance:
uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-zv3f"
I20240503 02:12:14.228914 10541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20240503 02:12:14.232584 10555 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.234198 10541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20240503 02:12:14.234366 10541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-zv3f"
I20240503 02:12:14.234606 10541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:14.277519 10405 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
I20240503 02:12:14.278159 10541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:14.279286 10541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:14.279611 10541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:14.280743 10541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:14.282533 10541 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:14.282624 10541 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.282774 10541 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:14.282853 10541 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.322255 10541 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.69:33471
I20240503 02:12:14.322309 10668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.69:33471 every 8 connection(s)
I20240503 02:12:14.323804 10541 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:14.329267 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 10541
I20240503 02:12:14.329514 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/instance
I20240503 02:12:14.333456 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.70:0
--local_ip_for_outbound_sockets=127.4.5.70
--webserver_interface=127.4.5.70
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:14.338923 10669 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:14.339306 10669 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:14.340360 10669 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:14.341895 9952 ts_manager.cc:194] Registered new tserver with Master: e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:14.342765 9952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.69:38167
W20240503 02:12:14.714449 10673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:14.714856 10673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:14.715025 10673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:14.723341 10673 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:14.723546 10673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:14.723743 10673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.70
I20240503 02:12:14.734068 10673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.70:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.4.5.70
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.70
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:14.735814 10673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:14.737973 10673 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:14.751773 10678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:14.752310 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:14.753046 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:14.753657 10673 server_base.cc:1034] running on GCE node
I20240503 02:12:14.754374 10673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:14.755529 10673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:14.756857 10673 hybrid_clock.cc:648] HybridClock initialized: now 1714702334756800 us; error 48 us; skew 500 ppm
I20240503 02:12:14.757244 10673 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:14.759945 10673 webserver.cc:460] Webserver started at http://127.4.5.70:34607/ using document root <none> and password file <none>
I20240503 02:12:14.760876 10673 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:14.761006 10673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:14.761356 10673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:14.764421 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/instance:
uuid: "8d454a3607d042dc9559ed59c3ccd84b"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-zv3f"
I20240503 02:12:14.765388 10673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/instance:
uuid: "8d454a3607d042dc9559ed59c3ccd84b"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-zv3f"
I20240503 02:12:14.771267 10673 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20240503 02:12:14.775061 10687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.776564 10673 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:12:14.776731 10673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8d454a3607d042dc9559ed59c3ccd84b"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-zv3f"
I20240503 02:12:14.776937 10673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:14.802906 10537 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
I20240503 02:12:14.805651 10673 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:14.806766 10673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:14.807087 10673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:14.808238 10673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:14.809980 10673 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:14.810086 10673 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.810209 10673 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:14.810287 10673 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:14.845839 10673 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.70:43707
I20240503 02:12:14.845877 10799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.70:43707 every 8 connection(s)
I20240503 02:12:14.847540 10673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:12:14.849623 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 10673
I20240503 02:12:14.849853 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/instance
I20240503 02:12:14.854303 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.71:0
--local_ip_for_outbound_sockets=127.4.5.71
--webserver_interface=127.4.5.71
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:14.865269 10800 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:14.865671 10800 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:14.866748 10800 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:14.868690 9952 ts_manager.cc:194] Registered new tserver with Master: 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:14.870008 9952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.70:36883
W20240503 02:12:15.245164 10804 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:15.245520 10804 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:15.245627 10804 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:15.254841 10804 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:15.255004 10804 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:15.255192 10804 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.71
I20240503 02:12:15.264828 10804 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.71:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.4.5.71
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.71
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:15.266377 10804 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:15.268357 10804 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:15.281641 10810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:15.282264 10809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:15.282830 10812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:15.283444 10804 server_base.cc:1034] running on GCE node
I20240503 02:12:15.284184 10804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:15.285238 10804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:15.286454 10804 hybrid_clock.cc:648] HybridClock initialized: now 1714702335286406 us; error 47 us; skew 500 ppm
I20240503 02:12:15.286787 10804 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:15.289381 10804 webserver.cc:460] Webserver started at http://127.4.5.71:45667/ using document root <none> and password file <none>
I20240503 02:12:15.290202 10804 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:15.290326 10804 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:15.290652 10804 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:15.293658 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/instance:
uuid: "f93b103af083471b8e8c0376e9148aad"
format_stamp: "Formatted at 2024-05-03 02:12:15 on dist-test-slave-zv3f"
I20240503 02:12:15.294489 10804 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/instance:
uuid: "f93b103af083471b8e8c0376e9148aad"
format_stamp: "Formatted at 2024-05-03 02:12:15 on dist-test-slave-zv3f"
I20240503 02:12:15.300029 10804 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20240503 02:12:15.303614 10818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:15.305075 10804 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:15.305229 10804 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f93b103af083471b8e8c0376e9148aad"
format_stamp: "Formatted at 2024-05-03 02:12:15 on dist-test-slave-zv3f"
I20240503 02:12:15.305440 10804 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:15.322911 10804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:15.324098 10804 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:15.324470 10804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:15.325735 10804 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:15.327785 10804 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:15.327906 10804 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:15.328011 10804 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:15.328088 10804 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:15.345746 10669 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
I20240503 02:12:15.365267 10804 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.71:46557
I20240503 02:12:15.365299 10931 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.71:46557 every 8 connection(s)
I20240503 02:12:15.366952 10804 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:12:15.373080 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 10804
I20240503 02:12:15.373306 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/instance
I20240503 02:12:15.377357 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.72:0
--local_ip_for_outbound_sockets=127.4.5.72
--webserver_interface=127.4.5.72
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:15.382968 10932 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:15.383352 10932 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:15.384549 10932 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:15.386569 9952 ts_manager.cc:194] Registered new tserver with Master: f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:15.387903 9952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.71:36181
W20240503 02:12:15.760882 10936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:15.761277 10936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:15.761422 10936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:15.769346 10936 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:15.769470 10936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:15.769676 10936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.72
I20240503 02:12:15.779021 10936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.72:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.4.5.72
--webserver_port=0
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.72
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:15.780632 10936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:15.782675 10936 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:15.796139 10941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:15.796900 10936 server_base.cc:1034] running on GCE node
W20240503 02:12:15.797350 10944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:15.798113 10942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:15.798825 10936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:15.799857 10936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:15.801074 10936 hybrid_clock.cc:648] HybridClock initialized: now 1714702335801018 us; error 45 us; skew 500 ppm
I20240503 02:12:15.801440 10936 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:15.804128 10936 webserver.cc:460] Webserver started at http://127.4.5.72:35929/ using document root <none> and password file <none>
I20240503 02:12:15.805038 10936 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:15.805187 10936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:15.805604 10936 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20240503 02:12:15.808919 10936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/instance:
uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
format_stamp: "Formatted at 2024-05-03 02:12:15 on dist-test-slave-zv3f"
I20240503 02:12:15.809876 10936 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/instance:
uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
format_stamp: "Formatted at 2024-05-03 02:12:15 on dist-test-slave-zv3f"
I20240503 02:12:15.815922 10936 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20240503 02:12:15.819669 10951 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:15.821194 10936 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20240503 02:12:15.821372 10936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
format_stamp: "Formatted at 2024-05-03 02:12:15 on dist-test-slave-zv3f"
I20240503 02:12:15.821586 10936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:15.873208 10800 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
I20240503 02:12:15.879560 10936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:15.880653 10936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:15.880940 10936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:15.882053 10936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:15.883764 10936 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:15.883873 10936 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:15.884004 10936 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:15.884078 10936 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:15.918576 10936 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.72:33671
I20240503 02:12:15.918628 11064 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.72:33671 every 8 connection(s)
I20240503 02:12:15.920197 10936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:15.925817 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 10936
I20240503 02:12:15.926016 4117 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/instance
I20240503 02:12:15.934706 11065 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:15.935031 11065 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:15.935909 11065 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:15.937394 9952 ts_manager.cc:194] Registered new tserver with Master: 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:15.938290 9952 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.72:34747
I20240503 02:12:15.942390 4117 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:12:15.959359 4117 external_mini_cluster.cc:930] 8 TS(s) registered with all masters
I20240503 02:12:15.959525 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 10541
I20240503 02:12:15.968851 10663 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:16.055254 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 10541
I20240503 02:12:16.072095 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 10673
I20240503 02:12:16.081487 10795 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:16.185631 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 10673
I20240503 02:12:16.202641 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 10804
I20240503 02:12:16.211633 10927 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:16.296198 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 10804
I20240503 02:12:16.317166 4117 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-task48YlKh/build/asan/bin/kudu pid 10936
I20240503 02:12:16.330262 11060 generic_service.cc:196] Checking for leaks (request via RPC)
I20240503 02:12:16.416874 4117 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task48YlKh/build/asan/bin/kudu with pid 10936
I20240503 02:12:20.186902 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:20.235319 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 16d0fa13022744c0a15d86cc341cb083 (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.237737 10471 tablet_service.cc:1467] Processing CreateTablet for tablet bb9d2573ecc1451894e2c685a9d71675 (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.238457 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d0fa13022744c0a15d86cc341cb083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.245034 10338 tablet_service.cc:1467] Processing CreateTablet for tablet f9b74c584e65451cb19fc004ac27f89a (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.246080 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 16d0fa13022744c0a15d86cc341cb083 (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.247149 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b74c584e65451cb19fc004ac27f89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.247841 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d0fa13022744c0a15d86cc341cb083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.252774 10207 tablet_service.cc:1467] Processing CreateTablet for tablet bb9d2573ecc1451894e2c685a9d71675 (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.245908 10206 tablet_service.cc:1467] Processing CreateTablet for tablet f9b74c584e65451cb19fc004ac27f89a (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.253463 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9d2573ecc1451894e2c685a9d71675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.255307 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b74c584e65451cb19fc004ac27f89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.259080 10339 tablet_service.cc:1467] Processing CreateTablet for tablet bb9d2573ecc1451894e2c685a9d71675 (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.259850 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9d2573ecc1451894e2c685a9d71675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.264799 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bb9d2573ecc1451894e2c685a9d71675. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.266799 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 16d0fa13022744c0a15d86cc341cb083 (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.269271 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16d0fa13022744c0a15d86cc341cb083. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.275242 10074 tablet_service.cc:1467] Processing CreateTablet for tablet f9b74c584e65451cb19fc004ac27f89a (DEFAULT_TABLE table=rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:20.276165 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b74c584e65451cb19fc004ac27f89a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:20.296859 11095 tablet_bootstrap.cc:492] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:20.301574 11095 tablet_bootstrap.cc:654] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.302837 11097 tablet_bootstrap.cc:492] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:20.303339 11096 tablet_bootstrap.cc:492] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:20.303316 11095 log.cc:824] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:20.307567 11096 tablet_bootstrap.cc:654] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.308187 11097 tablet_bootstrap.cc:654] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.309360 11096 log.cc:824] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:20.312597 11097 log.cc:824] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:20.314558 11100 tablet_bootstrap.cc:492] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:20.320964 11095 tablet_bootstrap.cc:492] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:20.321398 11095 ts_tablet_manager.cc:1367] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.025s user 0.016s sys 0.003s
I20240503 02:12:20.321725 11097 tablet_bootstrap.cc:492] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:20.321729 11096 tablet_bootstrap.cc:492] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:20.322065 11097 ts_tablet_manager.cc:1367] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.020s user 0.009s sys 0.000s
I20240503 02:12:20.322085 11096 ts_tablet_manager.cc:1367] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.019s user 0.012s sys 0.005s
I20240503 02:12:20.322455 11100 tablet_bootstrap.cc:654] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.324208 11100 log.cc:824] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:20.332720 11097 raft_consensus.cc:348] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.332720 11096 raft_consensus.cc:348] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.333400 11097 raft_consensus.cc:374] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.333400 11096 raft_consensus.cc:374] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.333531 11096 raft_consensus.cc:729] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.333541 11097 raft_consensus.cc:729] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.334232 11095 raft_consensus.cc:348] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:20.334565 11097 consensus_queue.cc:260] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.334558 11096 consensus_queue.cc:260] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.334816 11095 raft_consensus.cc:374] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.334939 11095 raft_consensus.cc:729] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.335964 11095 consensus_queue.cc:260] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:20.338565 11100 tablet_bootstrap.cc:492] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:20.339123 11100 ts_tablet_manager.cc:1367] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.000s
I20240503 02:12:20.346292 11095 ts_tablet_manager.cc:1398] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.025s user 0.017s sys 0.004s
I20240503 02:12:20.346909 11095 tablet_bootstrap.cc:492] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:20.349007 11095 tablet_bootstrap.cc:654] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.359124 11095 tablet_bootstrap.cc:492] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:20.359369 11095 ts_tablet_manager.cc:1367] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20240503 02:12:20.360819 11095 raft_consensus.cc:348] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.361119 11095 raft_consensus.cc:374] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.361223 11095 raft_consensus.cc:729] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.361573 11095 consensus_queue.cc:260] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.362069 11097 ts_tablet_manager.cc:1398] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.040s user 0.016s sys 0.022s
I20240503 02:12:20.362296 11095 ts_tablet_manager.cc:1398] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:20.362819 11097 tablet_bootstrap.cc:492] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:20.365572 11096 ts_tablet_manager.cc:1398] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.043s user 0.017s sys 0.020s
I20240503 02:12:20.366124 11096 tablet_bootstrap.cc:492] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:20.365769 11100 raft_consensus.cc:348] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.366286 11097 tablet_bootstrap.cc:654] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.366446 11100 raft_consensus.cc:374] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.366590 11100 raft_consensus.cc:729] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.367552 11100 consensus_queue.cc:260] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.368160 11096 tablet_bootstrap.cc:654] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.374065 11097 tablet_bootstrap.cc:492] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:20.374264 11097 ts_tablet_manager.cc:1367] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.004s
I20240503 02:12:20.374614 11096 tablet_bootstrap.cc:492] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:20.374789 11096 ts_tablet_manager.cc:1367] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.007s
I20240503 02:12:20.375303 11097 raft_consensus.cc:348] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.375515 11097 raft_consensus.cc:374] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.375624 11097 raft_consensus.cc:729] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.375898 11097 consensus_queue.cc:260] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.375859 11096 raft_consensus.cc:348] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.376107 11096 raft_consensus.cc:374] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.376228 11096 raft_consensus.cc:729] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.376549 11096 consensus_queue.cc:260] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.376881 11100 ts_tablet_manager.cc:1398] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.038s user 0.013s sys 0.008s
I20240503 02:12:20.377434 11100 tablet_bootstrap.cc:492] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:20.377843 11097 ts_tablet_manager.cc:1398] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:20.378934 11096 ts_tablet_manager.cc:1398] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:20.379299 11096 tablet_bootstrap.cc:492] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:20.381255 11096 tablet_bootstrap.cc:654] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.382100 11100 tablet_bootstrap.cc:654] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:20.385089 11096 tablet_bootstrap.cc:492] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:20.385200 11100 tablet_bootstrap.cc:492] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:20.385308 11096 ts_tablet_manager.cc:1367] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20240503 02:12:20.385378 11100 ts_tablet_manager.cc:1367] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 02:12:20.386200 11096 raft_consensus.cc:348] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:20.386487 11096 raft_consensus.cc:374] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.386605 11096 raft_consensus.cc:729] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.386535 11100 raft_consensus.cc:348] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:20.386770 11100 raft_consensus.cc:374] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:20.386862 11100 raft_consensus.cc:729] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:20.386899 11096 consensus_queue.cc:260] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:20.387123 11100 consensus_queue.cc:260] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:20.387595 11096 ts_tablet_manager.cc:1398] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:20.387699 11100 ts_tablet_manager.cc:1398] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:20.433902 11108 raft_consensus.cc:482] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.434337 11104 raft_consensus.cc:482] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.434310 11108 raft_consensus.cc:504] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.434649 11104 raft_consensus.cc:504] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:20.436800 11104 leader_election.cc:288] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:20.436779 11108 leader_election.cc:288] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:20.443030 11105 raft_consensus.cc:482] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:20.443497 11105 raft_consensus.cc:504] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.448572 11105 leader_election.cc:288] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:20.449237 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0fa13022744c0a15d86cc341cb083" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:20.450137 10491 raft_consensus.cc:2454] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:20.451164 10159 leader_election.cc:302] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:20.452001 11104 raft_consensus.cc:2781] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:20.452306 11104 raft_consensus.cc:482] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:20.452531 11104 raft_consensus.cc:3037] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.457724 11104 raft_consensus.cc:504] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
W20240503 02:12:20.458797 10141 tablet.cc:2369] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:20.461174 11104 leader_election.cc:288] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:20.461128 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0fa13022744c0a15d86cc341cb083" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:20.461486 10491 raft_consensus.cc:3037] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.465726 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b74c584e65451cb19fc004ac27f89a" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:20.466465 10227 raft_consensus.cc:2454] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:20.467396 10028 leader_election.cc:302] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:20.468021 11108 raft_consensus.cc:2781] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:20.468283 11108 raft_consensus.cc:482] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:20.468411 11108 raft_consensus.cc:3037] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.469349 10491 raft_consensus.cc:2454] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:20.470110 10159 leader_election.cc:302] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:20.470546 11104 raft_consensus.cc:2781] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:20.471149 11104 raft_consensus.cc:686] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:20.471846 11104 consensus_queue.cc:237] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:20.474005 11108 raft_consensus.cc:504] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.478500 11108 leader_election.cc:288] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:20.479084 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b74c584e65451cb19fc004ac27f89a" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:20.479382 10227 raft_consensus.cc:3037] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.479918 9949 catalog_manager.cc:5484] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:20.483006 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b74c584e65451cb19fc004ac27f89a" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:20.483609 10359 raft_consensus.cc:2454] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:20.485248 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b74c584e65451cb19fc004ac27f89a" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:20.485509 10359 raft_consensus.cc:3037] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.485869 10227 raft_consensus.cc:2454] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
W20240503 02:12:20.486351 10274 tablet.cc:2369] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:20.487545 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0fa13022744c0a15d86cc341cb083" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:20.487876 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16d0fa13022744c0a15d86cc341cb083" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:20.489022 10093 raft_consensus.cc:3037] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.489420 10359 raft_consensus.cc:2454] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:20.491434 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9d2573ecc1451894e2c685a9d71675" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:20.491808 10491 raft_consensus.cc:2454] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:20.492643 10292 leader_election.cc:302] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:20.493247 11105 raft_consensus.cc:2781] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:20.493412 11105 raft_consensus.cc:482] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:20.493525 11105 raft_consensus.cc:3037] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.495110 10093 raft_consensus.cc:2454] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:20.495393 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9d2573ecc1451894e2c685a9d71675" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:20.495791 10226 raft_consensus.cc:2454] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:20.487018 10028 leader_election.cc:302] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:20.497411 11105 raft_consensus.cc:504] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.498168 11108 raft_consensus.cc:2781] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:20.498891 11105 leader_election.cc:288] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:20.499044 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9d2573ecc1451894e2c685a9d71675" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:20.499346 10226 raft_consensus.cc:3037] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.499753 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bb9d2573ecc1451894e2c685a9d71675" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:20.500075 10491 raft_consensus.cc:3037] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:20.501627 11108 raft_consensus.cc:686] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:20.503279 10226 raft_consensus.cc:2454] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:20.503389 10491 raft_consensus.cc:2454] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:20.503072 11108 consensus_queue.cc:237] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.503943 10294 leader_election.cc:302] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:20.504348 11105 raft_consensus.cc:2781] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:20.505075 11105 raft_consensus.cc:686] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:20.505820 11105 consensus_queue.cc:237] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:20.510680 9949 catalog_manager.cc:5484] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:20.512064 9952 catalog_manager.cc:5484] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } }
W20240503 02:12:20.515049 10406 tablet.cc:2369] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:20.537194 10538 tablet.cc:2369] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:20.904981 11108 consensus_queue.cc:1035] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:20.912393 11108 consensus_queue.cc:1035] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:20.959383 11127 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:20.959831 11127 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:20.968429 11127 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:20.985150 11105 consensus_queue.cc:1035] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:20.992904 11105 consensus_queue.cc:1035] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_0'
I20240503 02:12:21.020412 11104 consensus_queue.cc:1035] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:21.031577 11104 consensus_queue.cc:1035] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 283.71 ms
time per row: 0.141855 ms
I20240503 02:12:21.434943 9947 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:21.445464 10339 tablet_service.cc:1467] Processing CreateTablet for tablet a2aac20a18f4444a9ff79642b23c89d9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.445771 10470 tablet_service.cc:1467] Processing CreateTablet for tablet a2aac20a18f4444a9ff79642b23c89d9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.446135 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2aac20a18f4444a9ff79642b23c89d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.446249 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2aac20a18f4444a9ff79642b23c89d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.446169 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 390ad47a9a534abb81d59dc7604c72d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.446554 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390ad47a9a534abb81d59dc7604c72d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.449456 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 390ad47a9a534abb81d59dc7604c72d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.449718 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 13e83ce8820b491992c041a8c21fa3df (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.449997 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390ad47a9a534abb81d59dc7604c72d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.449954 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 13e83ce8820b491992c041a8c21fa3df (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.450449 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e83ce8820b491992c041a8c21fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.451040 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e83ce8820b491992c041a8c21fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.452879 10074 tablet_service.cc:1467] Processing CreateTablet for tablet a2aac20a18f4444a9ff79642b23c89d9 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.453285 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2aac20a18f4444a9ff79642b23c89d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.457183 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 13e83ce8820b491992c041a8c21fa3df (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.457759 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e83ce8820b491992c041a8c21fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.461575 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 390ad47a9a534abb81d59dc7604c72d3 (DEFAULT_TABLE table=rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:21.462246 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390ad47a9a534abb81d59dc7604c72d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:21.469203 11199 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:21.471767 11198 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:21.473902 11199 tablet_bootstrap.cc:654] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.474073 11198 tablet_bootstrap.cc:654] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.475661 11201 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:21.477241 11200 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:21.479063 11198 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:21.479329 11201 tablet_bootstrap.cc:654] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.479353 11198 ts_tablet_manager.cc:1367] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20240503 02:12:21.479478 11200 tablet_bootstrap.cc:654] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.480579 11198 raft_consensus.cc:348] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.481011 11198 raft_consensus.cc:374] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.481165 11198 raft_consensus.cc:729] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.481472 11198 consensus_queue.cc:260] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.481981 11198 ts_tablet_manager.cc:1398] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:21.482036 11199 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:21.482234 11199 ts_tablet_manager.cc:1367] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.004s
I20240503 02:12:21.482278 11198 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:21.483786 11198 tablet_bootstrap.cc:654] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.484889 11199 raft_consensus.cc:348] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.485685 11199 raft_consensus.cc:374] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.485848 11199 raft_consensus.cc:729] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.486362 11201 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:21.486562 11201 ts_tablet_manager.cc:1367] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 02:12:21.486773 11199 consensus_queue.cc:260] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.487630 11201 raft_consensus.cc:348] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.487874 11201 raft_consensus.cc:374] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.487815 11199 ts_tablet_manager.cc:1398] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.005s user 0.000s sys 0.004s
I20240503 02:12:21.487967 11201 raft_consensus.cc:729] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.488353 11199 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:21.488286 11201 consensus_queue.cc:260] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.488941 11201 ts_tablet_manager.cc:1398] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:21.490820 11199 tablet_bootstrap.cc:654] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.492415 11200 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:21.492573 11198 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:21.492625 11200 ts_tablet_manager.cc:1367] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.004s
I20240503 02:12:21.492789 11198 ts_tablet_manager.cc:1367] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20240503 02:12:21.493290 11199 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:21.493458 11199 ts_tablet_manager.cc:1367] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:21.493525 11198 raft_consensus.cc:348] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.493749 11198 raft_consensus.cc:374] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.493636 11200 raft_consensus.cc:348] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.493829 11198 raft_consensus.cc:729] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.493875 11200 raft_consensus.cc:374] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.493966 11200 raft_consensus.cc:729] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.494136 11198 consensus_queue.cc:260] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.494242 11200 consensus_queue.cc:260] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.494299 11199 raft_consensus.cc:348] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.494501 11199 raft_consensus.cc:374] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.494568 11199 raft_consensus.cc:729] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.494864 11200 ts_tablet_manager.cc:1398] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:21.494856 11199 consensus_queue.cc:260] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.495309 11198 ts_tablet_manager.cc:1398] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:21.495369 11200 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:21.495615 11198 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:21.497154 11200 tablet_bootstrap.cc:654] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.496253 11199 ts_tablet_manager.cc:1398] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:21.498953 11198 tablet_bootstrap.cc:654] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.500310 11199 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:21.502912 11199 tablet_bootstrap.cc:654] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:21.504652 11198 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:21.504784 11198 ts_tablet_manager.cc:1367] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:12:21.504798 11200 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:21.504949 11200 ts_tablet_manager.cc:1367] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.001s
I20240503 02:12:21.505378 11199 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:21.505517 11199 ts_tablet_manager.cc:1367] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20240503 02:12:21.505595 11200 raft_consensus.cc:348] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.505661 11198 raft_consensus.cc:348] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.505798 11200 raft_consensus.cc:374] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.505868 11200 raft_consensus.cc:729] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.505874 11198 raft_consensus.cc:374] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.505973 11198 raft_consensus.cc:729] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.506186 11200 consensus_queue.cc:260] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.506290 11198 consensus_queue.cc:260] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.506453 11199 raft_consensus.cc:348] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.506639 11199 raft_consensus.cc:374] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:21.506718 11200 ts_tablet_manager.cc:1398] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:21.506740 11199 raft_consensus.cc:729] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:21.506875 11198 ts_tablet_manager.cc:1398] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:21.507006 11199 consensus_queue.cc:260] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.507969 11199 ts_tablet_manager.cc:1398] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:21.523526 11105 raft_consensus.cc:482] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.523788 11105 raft_consensus.cc:504] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.524667 11105 leader_election.cc:288] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:21.525352 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390ad47a9a534abb81d59dc7604c72d3" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:21.525652 10490 raft_consensus.cc:2454] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:21.526217 10292 leader_election.cc:302] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:21.526551 11105 raft_consensus.cc:2781] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:21.526665 11105 raft_consensus.cc:482] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:21.526799 11105 raft_consensus.cc:3037] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.529726 11105 raft_consensus.cc:504] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.530400 11105 leader_election.cc:288] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:21.530923 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390ad47a9a534abb81d59dc7604c72d3" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:21.531129 10490 raft_consensus.cc:3037] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.533051 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390ad47a9a534abb81d59dc7604c72d3" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:21.533373 10093 raft_consensus.cc:2454] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:21.533066 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390ad47a9a534abb81d59dc7604c72d3" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:21.533761 10094 raft_consensus.cc:3037] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.533932 10490 raft_consensus.cc:2454] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:21.534394 10292 leader_election.cc:302] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:21.534752 11105 raft_consensus.cc:2781] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:21.534929 11105 raft_consensus.cc:686] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:21.535233 11105 consensus_queue.cc:237] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.536635 10094 raft_consensus.cc:2454] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:21.538404 9952 catalog_manager.cc:5484] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:21.627959 11154 raft_consensus.cc:482] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.628244 11154 raft_consensus.cc:504] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.629186 11154 leader_election.cc:288] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:21.630151 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e83ce8820b491992c041a8c21fa3df" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:21.630504 10094 raft_consensus.cc:2454] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:21.630937 10161 leader_election.cc:302] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:21.631289 11154 raft_consensus.cc:2781] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:21.631443 11154 raft_consensus.cc:482] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:21.631563 11154 raft_consensus.cc:3037] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.634235 11154 raft_consensus.cc:504] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.635079 11154 leader_election.cc:288] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:21.635515 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e83ce8820b491992c041a8c21fa3df" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:21.635803 10094 raft_consensus.cc:3037] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.637436 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e83ce8820b491992c041a8c21fa3df" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:21.637635 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "13e83ce8820b491992c041a8c21fa3df" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:21.637868 10359 raft_consensus.cc:2454] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:21.638454 10094 raft_consensus.cc:2454] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:21.638804 10161 leader_election.cc:302] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:12:21.639142 11154 raft_consensus.cc:2781] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:21.639308 11154 raft_consensus.cc:686] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:21.639602 11154 consensus_queue.cc:237] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:21.642791 9952 catalog_manager.cc:5484] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:21.695310 11144 raft_consensus.cc:482] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:21.695614 11144 raft_consensus.cc:504] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.697539 11144 leader_election.cc:288] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:21.705065 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2aac20a18f4444a9ff79642b23c89d9" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:21.705369 10358 raft_consensus.cc:2454] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:21.705678 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2aac20a18f4444a9ff79642b23c89d9" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:21.705953 10094 raft_consensus.cc:2454] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:21.705919 10426 leader_election.cc:302] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:21.706350 11144 raft_consensus.cc:2781] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:21.706480 11144 raft_consensus.cc:482] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:21.706568 11144 raft_consensus.cc:3037] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.708847 11144 raft_consensus.cc:504] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.709434 11144 leader_election.cc:288] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:21.709892 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2aac20a18f4444a9ff79642b23c89d9" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:21.709935 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a2aac20a18f4444a9ff79642b23c89d9" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:21.710101 10358 raft_consensus.cc:3037] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.710163 10094 raft_consensus.cc:3037] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:21.712213 10358 raft_consensus.cc:2454] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:21.712236 10094 raft_consensus.cc:2454] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:21.712620 10426 leader_election.cc:302] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:21.712926 11144 raft_consensus.cc:2781] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:21.713372 11144 raft_consensus.cc:686] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:21.713951 11144 consensus_queue.cc:237] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:21.718156 9952 catalog_manager.cc:5484] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:22.031219 11105 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:22.038395 11105 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:22.161814 11219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:22.162246 11219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:22.170581 11219 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:22.186528 10358 raft_consensus.cc:3037] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.191520 11154 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:22.202270 11240 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_1'
I20240503 02:12:22.222496 11144 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:22.243887 11252 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 278.346 ms
time per row: 0.139173 ms
I20240503 02:12:22.640933 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:22.651818 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 48e4714b492d43af93050b4c07ebf0fc (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.652197 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 48e4714b492d43af93050b4c07ebf0fc (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.652738 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4714b492d43af93050b4c07ebf0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.652932 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 48e4714b492d43af93050b4c07ebf0fc (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.653146 10205 tablet_service.cc:1467] Processing CreateTablet for tablet e4806c1055824b67beb1643e3d42cb22 (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.653442 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4714b492d43af93050b4c07ebf0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.653543 10471 tablet_service.cc:1467] Processing CreateTablet for tablet e4806c1055824b67beb1643e3d42cb22 (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.653630 10339 tablet_service.cc:1467] Processing CreateTablet for tablet e4806c1055824b67beb1643e3d42cb22 (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.653980 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4806c1055824b67beb1643e3d42cb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.654111 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4806c1055824b67beb1643e3d42cb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.654390 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 48e4714b492d43af93050b4c07ebf0fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.654402 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4806c1055824b67beb1643e3d42cb22. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.664795 10074 tablet_service.cc:1467] Processing CreateTablet for tablet befeaccfcfe14e6bb51545a8290cea7a (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.665341 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befeaccfcfe14e6bb51545a8290cea7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.665277 10469 tablet_service.cc:1467] Processing CreateTablet for tablet befeaccfcfe14e6bb51545a8290cea7a (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.665875 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befeaccfcfe14e6bb51545a8290cea7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.667583 11286 tablet_bootstrap.cc:492] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:22.669585 11286 tablet_bootstrap.cc:654] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.673058 11289 tablet_bootstrap.cc:492] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:22.674448 10205 tablet_service.cc:1467] Processing CreateTablet for tablet befeaccfcfe14e6bb51545a8290cea7a (DEFAULT_TABLE table=rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:22.675067 11289 tablet_bootstrap.cc:654] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.675199 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet befeaccfcfe14e6bb51545a8290cea7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:22.676373 11287 tablet_bootstrap.cc:492] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:22.678444 11286 tablet_bootstrap.cc:492] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:22.678718 11286 ts_tablet_manager.cc:1367] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 02:12:22.679180 11287 tablet_bootstrap.cc:654] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.680083 11286 raft_consensus.cc:348] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.680476 11286 raft_consensus.cc:374] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.680646 11286 raft_consensus.cc:729] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.681026 11286 consensus_queue.cc:260] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.683219 11286 ts_tablet_manager.cc:1398] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:22.683568 11286 tablet_bootstrap.cc:492] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:22.684545 11287 tablet_bootstrap.cc:492] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:22.685207 11287 ts_tablet_manager.cc:1367] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.003s
I20240503 02:12:22.685859 11289 tablet_bootstrap.cc:492] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:22.686074 11289 ts_tablet_manager.cc:1367] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20240503 02:12:22.686450 11286 tablet_bootstrap.cc:654] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.687124 11289 raft_consensus.cc:348] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.687117 11287 raft_consensus.cc:348] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.687400 11289 raft_consensus.cc:374] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.687556 11289 raft_consensus.cc:729] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.688095 11287 raft_consensus.cc:374] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.688014 11289 consensus_queue.cc:260] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.688449 11287 raft_consensus.cc:729] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.688905 11289 ts_tablet_manager.cc:1398] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:22.689105 11287 consensus_queue.cc:260] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.690356 11287 ts_tablet_manager.cc:1398] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:22.692066 11289 tablet_bootstrap.cc:492] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:22.694324 11289 tablet_bootstrap.cc:654] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.698280 11238 raft_consensus.cc:482] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.698490 11238 raft_consensus.cc:504] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.699458 11238 leader_election.cc:288] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:22.700882 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4714b492d43af93050b4c07ebf0fc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:22.701752 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4714b492d43af93050b4c07ebf0fc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:22.701999 11290 tablet_bootstrap.cc:492] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
W20240503 02:12:22.703351 10161 leader_election.cc:341] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:22.703985 10159 leader_election.cc:341] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.704177 11290 tablet_bootstrap.cc:654] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.704139 10159 leader_election.cc:302] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eff8e8d11a4447218741bb994aa41082; no voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850
I20240503 02:12:22.704829 11238 raft_consensus.cc:2726] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:22.707458 11286 tablet_bootstrap.cc:492] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:22.707459 11289 tablet_bootstrap.cc:492] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:22.707643 11286 ts_tablet_manager.cc:1367] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.015s
I20240503 02:12:22.707652 11289 ts_tablet_manager.cc:1367] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.012s
I20240503 02:12:22.708097 11290 tablet_bootstrap.cc:492] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:22.708268 11290 ts_tablet_manager.cc:1367] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:12:22.708371 11286 raft_consensus.cc:348] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.708635 11286 raft_consensus.cc:374] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.708719 11286 raft_consensus.cc:729] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.708626 11289 raft_consensus.cc:348] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.708884 11289 raft_consensus.cc:374] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.708972 11289 raft_consensus.cc:729] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.709025 11286 consensus_queue.cc:260] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.709242 11289 consensus_queue.cc:260] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.709239 11290 raft_consensus.cc:348] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.709452 11290 raft_consensus.cc:374] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.709578 11290 raft_consensus.cc:729] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.709810 11286 ts_tablet_manager.cc:1398] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:22.709906 11289 ts_tablet_manager.cc:1398] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 02:12:22.709879 11290 consensus_queue.cc:260] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.710155 11286 tablet_bootstrap.cc:492] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:22.712078 11286 tablet_bootstrap.cc:654] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.713570 11290 ts_tablet_manager.cc:1398] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:12:22.713948 11290 tablet_bootstrap.cc:492] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:22.715008 11286 tablet_bootstrap.cc:492] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:22.715176 11286 ts_tablet_manager.cc:1367] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 02:12:22.716074 11286 raft_consensus.cc:348] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.716357 11286 raft_consensus.cc:374] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.716532 11286 raft_consensus.cc:729] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.716857 11286 consensus_queue.cc:260] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.717424 11286 ts_tablet_manager.cc:1398] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:22.718116 11290 tablet_bootstrap.cc:654] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.720367 11290 tablet_bootstrap.cc:492] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:22.720547 11290 ts_tablet_manager.cc:1367] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.002s
I20240503 02:12:22.721220 11290 raft_consensus.cc:348] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.721407 11290 raft_consensus.cc:374] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.721477 11290 raft_consensus.cc:729] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.721712 11290 consensus_queue.cc:260] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.722265 11290 ts_tablet_manager.cc:1398] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:22.722550 11290 tablet_bootstrap.cc:492] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:22.723202 11238 raft_consensus.cc:482] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.723345 11238 raft_consensus.cc:504] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.723974 11238 leader_election.cc:288] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:22.724309 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4806c1055824b67beb1643e3d42cb22" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:22.724702 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4806c1055824b67beb1643e3d42cb22" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
W20240503 02:12:22.724927 10159 leader_election.cc:341] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:22.725028 10358 raft_consensus.cc:2454] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:22.725100 11290 tablet_bootstrap.cc:654] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:22.725539 10161 leader_election.cc:302] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters: c6ec2f1283994a3089741715f811c850
I20240503 02:12:22.726048 11238 raft_consensus.cc:2781] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.726189 11238 raft_consensus.cc:482] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.726310 11238 raft_consensus.cc:3037] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.727226 11290 tablet_bootstrap.cc:492] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:22.727372 11290 ts_tablet_manager.cc:1367] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:22.728214 11290 raft_consensus.cc:348] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.728417 11290 raft_consensus.cc:374] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:22.728543 11290 raft_consensus.cc:729] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:22.728845 11290 consensus_queue.cc:260] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.729379 11238 raft_consensus.cc:504] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.729482 11290 ts_tablet_manager.cc:1398] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:22.730631 11238 leader_election.cc:288] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:22.730726 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4806c1055824b67beb1643e3d42cb22" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:22.730947 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4806c1055824b67beb1643e3d42cb22" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:22.731031 10358 raft_consensus.cc:3037] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.731133 10491 raft_consensus.cc:3037] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.733415 10358 raft_consensus.cc:2454] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:22.733734 10491 raft_consensus.cc:2454] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:22.733883 10161 leader_election.cc:302] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:22.734423 11238 raft_consensus.cc:2781] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.734583 11238 raft_consensus.cc:686] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:22.734962 11238 consensus_queue.cc:237] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:22.738298 9952 catalog_manager.cc:5484] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:22.763250 11144 raft_consensus.cc:482] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.763464 11144 raft_consensus.cc:504] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.764355 11144 leader_election.cc:288] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:22.765070 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4714b492d43af93050b4c07ebf0fc" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:22.765422 10093 raft_consensus.cc:2454] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:22.765978 10426 leader_election.cc:302] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:22.766263 11144 raft_consensus.cc:2781] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.766377 11144 raft_consensus.cc:482] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.766495 11144 raft_consensus.cc:3037] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.769280 11144 raft_consensus.cc:504] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.770126 11144 leader_election.cc:288] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:22.770576 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4714b492d43af93050b4c07ebf0fc" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:22.770833 10093 raft_consensus.cc:3037] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.772667 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4714b492d43af93050b4c07ebf0fc" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:22.772825 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "48e4714b492d43af93050b4c07ebf0fc" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:22.772931 10226 raft_consensus.cc:2454] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:22.773603 10093 raft_consensus.cc:2454] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:22.773960 10426 leader_election.cc:302] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters: eff8e8d11a4447218741bb994aa41082
I20240503 02:12:22.774282 11144 raft_consensus.cc:2781] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.774442 11144 raft_consensus.cc:686] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:22.774752 11144 consensus_queue.cc:237] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:22.778031 9952 catalog_manager.cc:5484] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:22.878917 11238 raft_consensus.cc:482] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:22.879149 11238 raft_consensus.cc:504] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.879825 11238 leader_election.cc:288] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:22.880338 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befeaccfcfe14e6bb51545a8290cea7a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:22.880388 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befeaccfcfe14e6bb51545a8290cea7a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:22.880632 10491 raft_consensus.cc:2454] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:22.880707 10093 raft_consensus.cc:2454] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:22.881074 10159 leader_election.cc:302] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:22.881356 11238 raft_consensus.cc:2781] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:22.881446 11238 raft_consensus.cc:482] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:22.881558 11238 raft_consensus.cc:3037] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.883661 11238 raft_consensus.cc:504] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.884228 11238 leader_election.cc:288] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:22.884737 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befeaccfcfe14e6bb51545a8290cea7a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:22.884760 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "befeaccfcfe14e6bb51545a8290cea7a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:22.884977 10491 raft_consensus.cc:3037] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.885004 10093 raft_consensus.cc:3037] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:22.886929 10491 raft_consensus.cc:2454] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:22.886991 10093 raft_consensus.cc:2454] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:22.887356 10159 leader_election.cc:302] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:22.887629 11238 raft_consensus.cc:2781] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:22.887776 11238 raft_consensus.cc:686] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:22.888095 11238 consensus_queue.cc:237] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:22.891954 9952 catalog_manager.cc:5484] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:23.146507 11238 consensus_queue.cc:1035] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:23.152653 11238 consensus_queue.cc:1035] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:23.246016 10227 raft_consensus.cc:3037] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:23.250684 11144 consensus_queue.cc:1035] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:23.256955 11252 consensus_queue.cc:1035] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:23.321321 11303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:23.321705 11303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:23.329743 11303 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_2'
I20240503 02:12:23.417915 11238 consensus_queue.cc:1035] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:23.423636 11240 consensus_queue.cc:1035] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 295.64 ms
time per row: 0.14782 ms
I20240503 02:12:23.781216 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:23.792078 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 476f6e414dae4db3ba9a187adad75f8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.792508 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 476f6e414dae4db3ba9a187adad75f8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.792897 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476f6e414dae4db3ba9a187adad75f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.793042 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476f6e414dae4db3ba9a187adad75f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.793388 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 36457442a8e24ece9a1c2d8cdab89b78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.793896 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36457442a8e24ece9a1c2d8cdab89b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.797668 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 36457442a8e24ece9a1c2d8cdab89b78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.798167 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36457442a8e24ece9a1c2d8cdab89b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.798547 10207 tablet_service.cc:1467] Processing CreateTablet for tablet d2fb90eef6224ed1a1dcc57c71683bc3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.799049 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fb90eef6224ed1a1dcc57c71683bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.801077 10471 tablet_service.cc:1467] Processing CreateTablet for tablet d2fb90eef6224ed1a1dcc57c71683bc3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.801669 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fb90eef6224ed1a1dcc57c71683bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.803565 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 476f6e414dae4db3ba9a187adad75f8c (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.804064 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476f6e414dae4db3ba9a187adad75f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.805662 10337 tablet_service.cc:1467] Processing CreateTablet for tablet d2fb90eef6224ed1a1dcc57c71683bc3 (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.806314 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fb90eef6224ed1a1dcc57c71683bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.808636 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 36457442a8e24ece9a1c2d8cdab89b78 (DEFAULT_TABLE table=rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:23.809221 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36457442a8e24ece9a1c2d8cdab89b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:23.813409 11369 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:23.813731 11368 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:23.816576 11369 tablet_bootstrap.cc:654] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.816601 11368 tablet_bootstrap.cc:654] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.819996 11370 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:23.822031 11370 tablet_bootstrap.cc:654] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.825181 11367 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:23.829334 11367 tablet_bootstrap.cc:654] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.842995 11370 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:23.843397 11370 ts_tablet_manager.cc:1367] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.011s
I20240503 02:12:23.844962 11370 raft_consensus.cc:348] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:23.845513 11370 raft_consensus.cc:374] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.845697 11370 raft_consensus.cc:729] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.846174 11370 consensus_queue.cc:260] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:23.848136 11370 ts_tablet_manager.cc:1398] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:23.848587 11370 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:23.850925 11370 tablet_bootstrap.cc:654] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.862588 11369 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:23.862588 11367 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:23.862833 11367 ts_tablet_manager.cc:1367] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.038s user 0.004s sys 0.007s
I20240503 02:12:23.862848 11369 ts_tablet_manager.cc:1367] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.050s user 0.025s sys 0.014s
I20240503 02:12:23.863822 11367 raft_consensus.cc:348] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:23.863823 11369 raft_consensus.cc:348] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:23.864080 11367 raft_consensus.cc:374] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.864086 11369 raft_consensus.cc:374] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.864190 11369 raft_consensus.cc:729] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.864199 11367 raft_consensus.cc:729] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.864523 11369 consensus_queue.cc:260] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:23.864497 11367 consensus_queue.cc:260] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:23.865131 11367 ts_tablet_manager.cc:1398] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:23.865190 11369 ts_tablet_manager.cc:1398] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:23.865464 11367 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:23.865494 11369 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:23.867492 11367 tablet_bootstrap.cc:654] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.870548 11368 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:23.870785 11368 ts_tablet_manager.cc:1367] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.057s user 0.004s sys 0.009s
I20240503 02:12:23.872774 11370 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:23.873086 11370 ts_tablet_manager.cc:1367] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.025s user 0.018s sys 0.003s
I20240503 02:12:23.874179 11370 raft_consensus.cc:348] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:23.874593 11370 raft_consensus.cc:374] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.874742 11370 raft_consensus.cc:729] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.875164 11370 consensus_queue.cc:260] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:23.875568 11369 tablet_bootstrap.cc:654] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.876855 11370 ts_tablet_manager.cc:1398] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:23.876762 11368 raft_consensus.cc:348] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:23.877066 11368 raft_consensus.cc:374] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.877188 11368 raft_consensus.cc:729] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.877511 11368 consensus_queue.cc:260] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:23.878157 11368 ts_tablet_manager.cc:1398] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.007s user 0.004s sys 0.000s
I20240503 02:12:23.878513 11368 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:23.878777 11369 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:23.878996 11369 ts_tablet_manager.cc:1367] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20240503 02:12:23.882800 11369 raft_consensus.cc:348] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:23.883074 11369 raft_consensus.cc:374] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.883154 11369 raft_consensus.cc:729] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.883265 11367 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:23.883551 11367 ts_tablet_manager.cc:1367] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.009s
I20240503 02:12:23.883442 11369 consensus_queue.cc:260] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:23.884027 11369 ts_tablet_manager.cc:1398] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:23.885463 11367 raft_consensus.cc:348] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:23.885797 11367 raft_consensus.cc:374] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.885950 11367 raft_consensus.cc:729] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.886014 11368 tablet_bootstrap.cc:654] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.886379 11367 consensus_queue.cc:260] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:23.887718 11367 ts_tablet_manager.cc:1398] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:23.888062 11367 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:23.889192 11368 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:23.889376 11368 ts_tablet_manager.cc:1367] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 02:12:23.889791 11367 tablet_bootstrap.cc:654] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:23.890430 11368 raft_consensus.cc:348] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:23.890712 11368 raft_consensus.cc:374] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.890808 11368 raft_consensus.cc:729] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.891100 11368 consensus_queue.cc:260] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:23.891830 11368 ts_tablet_manager.cc:1398] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:23.892414 11367 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:23.892664 11367 ts_tablet_manager.cc:1367] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20240503 02:12:23.893370 11367 raft_consensus.cc:348] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:23.893553 11367 raft_consensus.cc:374] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:23.893658 11367 raft_consensus.cc:729] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:23.893895 11367 consensus_queue.cc:260] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:23.894496 11367 ts_tablet_manager.cc:1398] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:24.064018 11350 raft_consensus.cc:482] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.064234 11350 raft_consensus.cc:504] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:24.064993 11350 leader_election.cc:288] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:24.065559 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2fb90eef6224ed1a1dcc57c71683bc3" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:24.065587 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2fb90eef6224ed1a1dcc57c71683bc3" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:24.065809 10358 raft_consensus.cc:2454] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:24.065877 10490 raft_consensus.cc:2454] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:24.066303 10161 leader_election.cc:302] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:24.066606 11350 raft_consensus.cc:2781] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.066725 11350 raft_consensus.cc:482] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.066809 11350 raft_consensus.cc:3037] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.068924 11350 raft_consensus.cc:504] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:24.069483 11350 leader_election.cc:288] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:24.069980 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2fb90eef6224ed1a1dcc57c71683bc3" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:24.069998 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2fb90eef6224ed1a1dcc57c71683bc3" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:24.070190 10358 raft_consensus.cc:3037] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.070205 10490 raft_consensus.cc:3037] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.072116 10490 raft_consensus.cc:2454] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:24.072348 10358 raft_consensus.cc:2454] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:24.072535 10159 leader_election.cc:302] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:24.072851 11350 raft_consensus.cc:2781] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.072984 11350 raft_consensus.cc:686] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:24.073259 11350 consensus_queue.cc:237] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:24.076217 9948 catalog_manager.cc:5484] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:24.111655 11122 raft_consensus.cc:482] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.111932 11122 raft_consensus.cc:504] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:24.112754 11122 leader_election.cc:288] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:24.113339 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476f6e414dae4db3ba9a187adad75f8c" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:24.113517 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476f6e414dae4db3ba9a187adad75f8c" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:24.113637 10358 raft_consensus.cc:2454] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:24.113823 10227 raft_consensus.cc:2454] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:24.114230 10028 leader_election.cc:302] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:24.114514 11122 raft_consensus.cc:2781] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.114666 11122 raft_consensus.cc:482] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.114764 11122 raft_consensus.cc:3037] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.117211 11122 raft_consensus.cc:504] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:24.117885 11122 leader_election.cc:288] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:24.118367 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476f6e414dae4db3ba9a187adad75f8c" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:24.118501 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "476f6e414dae4db3ba9a187adad75f8c" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:24.118633 10358 raft_consensus.cc:3037] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.118760 10227 raft_consensus.cc:3037] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.121706 10227 raft_consensus.cc:2454] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:24.121726 10358 raft_consensus.cc:2454] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:24.122112 10028 leader_election.cc:302] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:24.122416 11122 raft_consensus.cc:2781] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.122829 11122 raft_consensus.cc:686] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:24.123162 11122 consensus_queue.cc:237] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:24.126096 9948 catalog_manager.cc:5484] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:24.239866 11122 raft_consensus.cc:482] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:24.240147 11122 raft_consensus.cc:504] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:24.241061 11122 leader_election.cc:288] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:24.241773 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:24.242134 10358 raft_consensus.cc:2454] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:24.242616 10028 leader_election.cc:302] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:24.242965 11122 raft_consensus.cc:2781] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:24.243072 11122 raft_consensus.cc:482] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:24.243196 11122 raft_consensus.cc:3037] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.246038 11122 raft_consensus.cc:504] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:24.247167 11122 leader_election.cc:288] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:24.247328 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:24.247570 10358 raft_consensus.cc:3037] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.248560 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:24.248804 10490 raft_consensus.cc:2454] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:24.248744 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:24.248955 10489 raft_consensus.cc:3037] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:24.250434 10358 raft_consensus.cc:2454] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:24.250869 10028 leader_election.cc:302] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:24.251149 11122 raft_consensus.cc:2781] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:24.251340 11122 raft_consensus.cc:686] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:24.251750 11122 consensus_queue.cc:237] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:24.251945 10489 raft_consensus.cc:2454] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:24.255103 9948 catalog_manager.cc:5484] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:24.537569 11122 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:24.545249 11122 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:24.613785 11350 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:24.620944 11350 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:24.682500 11387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:24.682948 11387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:24.691407 11387 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:24.693372 11122 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:24.700706 11382 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_3'
W20240503 02:12:24.940405 10161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.65:38119, user_credentials={real_user=slave}} blocked reactor thread for 47232us
INSERT report
rows total: 2000
time total: 307.855 ms
time per row: 0.153928 ms
I20240503 02:12:25.167819 9943 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:25.178145 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 38b1d32c5ca040fcb67aaad68aef60f2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.178303 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 38b1d32c5ca040fcb67aaad68aef60f2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.178450 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 38b1d32c5ca040fcb67aaad68aef60f2 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.178819 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b1d32c5ca040fcb67aaad68aef60f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.178799 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b1d32c5ca040fcb67aaad68aef60f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.178891 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38b1d32c5ca040fcb67aaad68aef60f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.178794 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 6441ebd07bac41e6ae31cbb7e95c67af (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.179497 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6441ebd07bac41e6ae31cbb7e95c67af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.183320 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 6441ebd07bac41e6ae31cbb7e95c67af (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.183758 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6441ebd07bac41e6ae31cbb7e95c67af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.185111 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 6441ebd07bac41e6ae31cbb7e95c67af (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.185632 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6441ebd07bac41e6ae31cbb7e95c67af. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.189133 10469 tablet_service.cc:1467] Processing CreateTablet for tablet ee51997a3a0341719b63934754f13564 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.189684 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee51997a3a0341719b63934754f13564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.190191 10338 tablet_service.cc:1467] Processing CreateTablet for tablet ee51997a3a0341719b63934754f13564 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.190656 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee51997a3a0341719b63934754f13564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.190866 10072 tablet_service.cc:1467] Processing CreateTablet for tablet ee51997a3a0341719b63934754f13564 (DEFAULT_TABLE table=rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:25.191313 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee51997a3a0341719b63934754f13564. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:25.216717 11455 tablet_bootstrap.cc:492] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:25.218515 11456 tablet_bootstrap.cc:492] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:25.221450 11457 tablet_bootstrap.cc:492] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:25.221782 11455 tablet_bootstrap.cc:654] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.223752 11457 tablet_bootstrap.cc:654] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.225903 11456 tablet_bootstrap.cc:654] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.229377 11454 tablet_bootstrap.cc:492] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:25.231755 11456 tablet_bootstrap.cc:492] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:25.231938 11456 ts_tablet_manager.cc:1367] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240503 02:12:25.232422 11454 tablet_bootstrap.cc:654] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.232985 11456 raft_consensus.cc:348] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.233258 11456 raft_consensus.cc:374] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.233350 11456 raft_consensus.cc:729] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.233666 11456 consensus_queue.cc:260] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.234440 11456 ts_tablet_manager.cc:1398] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20240503 02:12:25.234912 11456 tablet_bootstrap.cc:492] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:25.236675 11456 tablet_bootstrap.cc:654] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.241609 11457 tablet_bootstrap.cc:492] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:25.241967 11457 ts_tablet_manager.cc:1367] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20240503 02:12:25.244939 11454 tablet_bootstrap.cc:492] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:25.245126 11454 ts_tablet_manager.cc:1367] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.004s
I20240503 02:12:25.244936 11457 raft_consensus.cc:348] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.245275 11456 tablet_bootstrap.cc:492] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:25.245450 11457 raft_consensus.cc:374] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.245561 11456 ts_tablet_manager.cc:1367] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20240503 02:12:25.245592 11457 raft_consensus.cc:729] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.246078 11457 consensus_queue.cc:260] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.246179 11454 raft_consensus.cc:348] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.246418 11454 raft_consensus.cc:374] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.246519 11454 raft_consensus.cc:729] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.246770 11454 consensus_queue.cc:260] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.246922 11456 raft_consensus.cc:348] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.247260 11456 raft_consensus.cc:374] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.247359 11456 raft_consensus.cc:729] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.247736 11456 consensus_queue.cc:260] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.247469 11454 ts_tablet_manager.cc:1398] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:25.248471 11454 tablet_bootstrap.cc:492] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:25.250399 11454 tablet_bootstrap.cc:654] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.252269 11456 ts_tablet_manager.cc:1398] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240503 02:12:25.255860 11456 tablet_bootstrap.cc:492] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:25.258412 11456 tablet_bootstrap.cc:654] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.259526 11454 tablet_bootstrap.cc:492] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:25.259708 11454 ts_tablet_manager.cc:1367] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20240503 02:12:25.260560 11454 raft_consensus.cc:348] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.260847 11454 raft_consensus.cc:374] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.260921 11454 raft_consensus.cc:729] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.261204 11454 consensus_queue.cc:260] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.261948 11454 ts_tablet_manager.cc:1398] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:25.265117 11457 ts_tablet_manager.cc:1398] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 02:12:25.266314 11457 tablet_bootstrap.cc:492] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:25.267817 11456 tablet_bootstrap.cc:492] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:25.267995 11456 ts_tablet_manager.cc:1367] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.009s
I20240503 02:12:25.268474 11457 tablet_bootstrap.cc:654] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.268978 11456 raft_consensus.cc:348] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.269228 11456 raft_consensus.cc:374] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.269337 11456 raft_consensus.cc:729] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.270335 11456 consensus_queue.cc:260] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.270937 11456 ts_tablet_manager.cc:1398] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:25.277201 11455 tablet_bootstrap.cc:492] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:25.277992 11455 ts_tablet_manager.cc:1367] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.061s user 0.024s sys 0.023s
I20240503 02:12:25.278442 11457 tablet_bootstrap.cc:492] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:25.278682 11457 ts_tablet_manager.cc:1367] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.008s
I20240503 02:12:25.279018 11455 raft_consensus.cc:348] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.279259 11455 raft_consensus.cc:374] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.279383 11455 raft_consensus.cc:729] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.279667 11455 consensus_queue.cc:260] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.279703 11457 raft_consensus.cc:348] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.280041 11457 raft_consensus.cc:374] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.280139 11457 raft_consensus.cc:729] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.280486 11457 consensus_queue.cc:260] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.282301 11457 ts_tablet_manager.cc:1398] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:25.282563 11455 ts_tablet_manager.cc:1398] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:25.284940 11455 tablet_bootstrap.cc:492] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:25.286912 11455 tablet_bootstrap.cc:654] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:25.288969 11455 tablet_bootstrap.cc:492] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:25.289119 11455 ts_tablet_manager.cc:1367] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:12:25.289849 11455 raft_consensus.cc:348] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.290074 11455 raft_consensus.cc:374] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:25.290148 11455 raft_consensus.cc:729] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:25.290378 11455 consensus_queue.cc:260] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.290925 11455 ts_tablet_manager.cc:1398] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:25.413684 11105 raft_consensus.cc:482] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.413895 11105 raft_consensus.cc:504] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.414678 11105 leader_election.cc:288] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:25.415124 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b1d32c5ca040fcb67aaad68aef60f2" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:25.415419 10093 raft_consensus.cc:2454] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:25.415364 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b1d32c5ca040fcb67aaad68aef60f2" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:25.415624 10227 raft_consensus.cc:2454] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:25.416128 10294 leader_election.cc:302] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:25.416396 11105 raft_consensus.cc:2781] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.416550 11105 raft_consensus.cc:482] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.416637 11105 raft_consensus.cc:3037] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.418764 11105 raft_consensus.cc:504] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.419322 11105 leader_election.cc:288] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:25.419821 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b1d32c5ca040fcb67aaad68aef60f2" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:25.419847 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38b1d32c5ca040fcb67aaad68aef60f2" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:25.420054 10093 raft_consensus.cc:3037] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.420056 10227 raft_consensus.cc:3037] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.423048 10227 raft_consensus.cc:2454] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:25.423048 10093 raft_consensus.cc:2454] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:25.423449 10294 leader_election.cc:302] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:25.423743 11105 raft_consensus.cc:2781] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.424171 11105 raft_consensus.cc:686] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:25.424474 11105 consensus_queue.cc:237] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.427702 9951 catalog_manager.cc:5484] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:25.459925 11346 raft_consensus.cc:482] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.460160 11346 raft_consensus.cc:504] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.460882 11346 leader_election.cc:288] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:25.461508 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee51997a3a0341719b63934754f13564" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:25.461591 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee51997a3a0341719b63934754f13564" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:25.461793 10359 raft_consensus.cc:2454] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:25.461863 10093 raft_consensus.cc:2454] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:25.462292 10426 leader_election.cc:302] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:25.462568 11346 raft_consensus.cc:2781] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.462671 11346 raft_consensus.cc:482] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.462778 11346 raft_consensus.cc:3037] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.464833 11346 raft_consensus.cc:504] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.465330 11346 leader_election.cc:288] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:25.465728 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee51997a3a0341719b63934754f13564" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:25.465801 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee51997a3a0341719b63934754f13564" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:25.465938 10359 raft_consensus.cc:3037] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.466028 10093 raft_consensus.cc:3037] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.467865 10093 raft_consensus.cc:2454] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:25.467864 10359 raft_consensus.cc:2454] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:25.468268 10426 leader_election.cc:302] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:25.468619 11346 raft_consensus.cc:2781] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.469022 11346 raft_consensus.cc:686] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:25.469349 11346 consensus_queue.cc:237] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:25.472402 9951 catalog_manager.cc:5484] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:25.484999 11433 raft_consensus.cc:482] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:25.485220 11433 raft_consensus.cc:504] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.486047 11433 leader_election.cc:288] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:25.486603 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6441ebd07bac41e6ae31cbb7e95c67af" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:25.486589 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6441ebd07bac41e6ae31cbb7e95c67af" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:25.486842 10093 raft_consensus.cc:2454] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:25.486837 10490 raft_consensus.cc:2454] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:25.487268 10161 leader_election.cc:302] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:25.487546 11433 raft_consensus.cc:2781] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:25.487668 11433 raft_consensus.cc:482] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:25.487746 11433 raft_consensus.cc:3037] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.489799 11433 raft_consensus.cc:504] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.490458 11433 leader_election.cc:288] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:25.490888 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6441ebd07bac41e6ae31cbb7e95c67af" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:25.490970 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6441ebd07bac41e6ae31cbb7e95c67af" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:25.491093 10093 raft_consensus.cc:3037] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.491194 10490 raft_consensus.cc:3037] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:25.493510 10093 raft_consensus.cc:2454] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:25.493557 10490 raft_consensus.cc:2454] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:25.493974 10161 leader_election.cc:302] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:25.494280 11433 raft_consensus.cc:2781] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:25.494441 11433 raft_consensus.cc:686] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:25.494733 11433 consensus_queue.cc:237] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:25.498040 9951 catalog_manager.cc:5484] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } }
W20240503 02:12:25.874483 11472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:25.874882 11472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:25.882441 11472 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_4'
I20240503 02:12:25.931816 11105 consensus_queue.cc:1035] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.941201 11105 consensus_queue.cc:1035] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.944607 11433 consensus_queue.cc:1035] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.968782 11394 consensus_queue.cc:1035] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:25.999146 11346 consensus_queue.cc:1035] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:26.024053 11346 consensus_queue.cc:1035] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
INSERT report
rows total: 2000
time total: 299.57 ms
time per row: 0.149785 ms
-----------------------------------------------------
Suppressions used:
count bytes template
40 10276 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:12:27.318634 9943 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:27.329106 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5125fbc8ec4ba887abce94a84efbdc (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.329362 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5125fbc8ec4ba887abce94a84efbdc (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.329766 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5125fbc8ec4ba887abce94a84efbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.329896 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5125fbc8ec4ba887abce94a84efbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.329883 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8b006a679a4b8d8d3551310b78fb3a (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.330264 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8b006a679a4b8d8d3551310b78fb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.332137 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 0a5125fbc8ec4ba887abce94a84efbdc (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.332966 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8b006a679a4b8d8d3551310b78fb3a (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.333981 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5125fbc8ec4ba887abce94a84efbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.334260 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.334635 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.334698 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 9f8b006a679a4b8d8d3551310b78fb3a (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.335098 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8b006a679a4b8d8d3551310b78fb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.336898 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8b006a679a4b8d8d3551310b78fb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.340941 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.341406 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.345036 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 (DEFAULT_TABLE table=rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:27.345588 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:27.354997 11545 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:27.357810 11545 tablet_bootstrap.cc:654] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.360395 11546 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:27.364547 11546 tablet_bootstrap.cc:654] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.366667 11544 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:27.368842 11544 tablet_bootstrap.cc:654] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.369293 11549 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:27.370581 11546 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:27.370911 11546 ts_tablet_manager.cc:1367] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 02:12:27.372188 11546 raft_consensus.cc:348] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.372781 11546 raft_consensus.cc:374] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.372968 11546 raft_consensus.cc:729] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.373068 11549 tablet_bootstrap.cc:654] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.373407 11546 consensus_queue.cc:260] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.374339 11546 ts_tablet_manager.cc:1398] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:27.375600 11546 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:27.376641 11544 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:27.376649 11545 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:27.376832 11544 ts_tablet_manager.cc:1367] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20240503 02:12:27.376832 11545 ts_tablet_manager.cc:1367] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.022s user 0.011s sys 0.007s
I20240503 02:12:27.377604 11546 tablet_bootstrap.cc:654] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.377768 11545 raft_consensus.cc:348] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.377772 11544 raft_consensus.cc:348] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.378023 11545 raft_consensus.cc:374] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.378023 11544 raft_consensus.cc:374] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.378116 11545 raft_consensus.cc:729] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.378423 11545 consensus_queue.cc:260] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.379201 11545 ts_tablet_manager.cc:1398] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:27.379534 11545 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:27.379869 11544 raft_consensus.cc:729] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.380247 11544 consensus_queue.cc:260] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.381085 11544 ts_tablet_manager.cc:1398] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:27.381425 11544 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:27.382427 11545 tablet_bootstrap.cc:654] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.383412 11544 tablet_bootstrap.cc:654] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.386960 11545 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:27.387210 11545 ts_tablet_manager.cc:1367] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.004s
I20240503 02:12:27.389662 11545 raft_consensus.cc:348] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.390013 11545 raft_consensus.cc:374] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.390102 11545 raft_consensus.cc:729] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.390996 11545 consensus_queue.cc:260] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.392884 11545 ts_tablet_manager.cc:1398] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20240503 02:12:27.394488 11544 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:27.394714 11544 ts_tablet_manager.cc:1367] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.000s
I20240503 02:12:27.395426 11546 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:27.395630 11546 ts_tablet_manager.cc:1367] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.011s
I20240503 02:12:27.395627 11544 raft_consensus.cc:348] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.395877 11544 raft_consensus.cc:374] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.395957 11544 raft_consensus.cc:729] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.396294 11544 consensus_queue.cc:260] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.396430 11549 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:27.396683 11522 raft_consensus.cc:482] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.396723 11549 ts_tablet_manager.cc:1367] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.010s
I20240503 02:12:27.396590 11546 raft_consensus.cc:348] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.396854 11546 raft_consensus.cc:374] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.396950 11546 raft_consensus.cc:729] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.397218 11546 consensus_queue.cc:260] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.396873 11522 raft_consensus.cc:504] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.397866 11549 raft_consensus.cc:348] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.398146 11549 raft_consensus.cc:374] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.398278 11549 raft_consensus.cc:729] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.398625 11522 leader_election.cc:288] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:27.399904 11544 ts_tablet_manager.cc:1398] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:27.398550 11549 consensus_queue.cc:260] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.400415 11544 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:27.400519 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5125fbc8ec4ba887abce94a84efbdc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:27.401036 10489 raft_consensus.cc:2454] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:27.401953 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5125fbc8ec4ba887abce94a84efbdc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:27.402274 10358 raft_consensus.cc:2454] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:27.402779 11544 tablet_bootstrap.cc:654] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.403329 11504 raft_consensus.cc:482] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.403532 11504 raft_consensus.cc:504] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.404546 11504 leader_election.cc:288] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:27.407979 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fadc7e9c7604b5d8f7b8751b6ca78a7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
W20240503 02:12:27.410671 10028 leader_election.cc:341] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:27.411624 11544 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:27.411677 10159 leader_election.cc:302] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:27.412034 11544 ts_tablet_manager.cc:1367] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.011s
I20240503 02:12:27.413276 11544 raft_consensus.cc:348] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.413672 11544 raft_consensus.cc:374] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.413846 11544 raft_consensus.cc:729] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.414301 11544 consensus_queue.cc:260] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.416550 11522 raft_consensus.cc:2781] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.416712 11522 raft_consensus.cc:482] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.416811 11522 raft_consensus.cc:3037] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.418313 11544 ts_tablet_manager.cc:1398] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:27.414918 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fadc7e9c7604b5d8f7b8751b6ca78a7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:27.418995 10358 raft_consensus.cc:2454] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:27.419677 10028 leader_election.cc:302] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters: eff8e8d11a4447218741bb994aa41082
I20240503 02:12:27.420423 11504 raft_consensus.cc:2781] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.420627 11504 raft_consensus.cc:482] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.420751 11504 raft_consensus.cc:3037] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.426321 11522 raft_consensus.cc:504] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.427981 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5125fbc8ec4ba887abce94a84efbdc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:27.428308 10358 raft_consensus.cc:3037] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.436658 11546 ts_tablet_manager.cc:1398] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.041s user 0.016s sys 0.023s
I20240503 02:12:27.436902 11549 ts_tablet_manager.cc:1398] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.040s user 0.017s sys 0.020s
I20240503 02:12:27.437083 11546 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:27.438805 10358 raft_consensus.cc:2454] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:27.439042 11546 tablet_bootstrap.cc:654] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:27.439481 11504 raft_consensus.cc:504] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.439610 10161 leader_election.cc:302] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:27.441267 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fadc7e9c7604b5d8f7b8751b6ca78a7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:27.441622 10358 raft_consensus.cc:3037] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.442884 11522 leader_election.cc:288] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:27.443219 11522 raft_consensus.cc:2781] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:27.444078 11504 leader_election.cc:288] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:27.445308 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fadc7e9c7604b5d8f7b8751b6ca78a7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
W20240503 02:12:27.446277 10028 leader_election.cc:341] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:27.448045 10358 raft_consensus.cc:2454] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:27.448626 10028 leader_election.cc:302] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters: eff8e8d11a4447218741bb994aa41082
I20240503 02:12:27.448979 11504 raft_consensus.cc:2781] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:27.449141 11504 raft_consensus.cc:686] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:27.449100 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0a5125fbc8ec4ba887abce94a84efbdc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:27.449285 11522 raft_consensus.cc:686] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:27.449417 10489 raft_consensus.cc:3037] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.449467 11504 consensus_queue.cc:237] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.449585 11522 consensus_queue.cc:237] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:27.452217 10489 raft_consensus.cc:2454] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:27.453476 11546 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:27.453653 11546 ts_tablet_manager.cc:1367] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.009s
I20240503 02:12:27.454388 11546 raft_consensus.cc:348] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.454576 11546 raft_consensus.cc:374] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:27.454682 11546 raft_consensus.cc:729] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:27.454913 11546 consensus_queue.cc:260] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.455621 11546 ts_tablet_manager.cc:1398] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:27.457394 9943 catalog_manager.cc:5484] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:27.459436 9951 catalog_manager.cc:5484] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:27.489872 11522 raft_consensus.cc:482] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:27.490204 11522 raft_consensus.cc:504] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.491101 11522 leader_election.cc:288] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:27.491796 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:27.491820 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:27.492064 10489 raft_consensus.cc:2454] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:27.492094 10358 raft_consensus.cc:2454] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:27.492559 10159 leader_election.cc:302] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:27.492964 11522 raft_consensus.cc:2781] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:27.493055 11522 raft_consensus.cc:482] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:27.493170 11522 raft_consensus.cc:3037] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.514642 11522 raft_consensus.cc:504] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.515506 11522 leader_election.cc:288] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:27.516216 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:27.516233 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:27.516500 10489 raft_consensus.cc:3037] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.516526 10358 raft_consensus.cc:3037] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.519681 10358 raft_consensus.cc:2454] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:27.519703 10489 raft_consensus.cc:2454] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:27.520220 10159 leader_election.cc:302] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:27.520577 11522 raft_consensus.cc:2781] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:27.520753 11522 raft_consensus.cc:686] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:27.521071 11522 consensus_queue.cc:237] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:27.524765 9951 catalog_manager.cc:5484] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:27.832541 11504 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.840842 10227 raft_consensus.cc:3037] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:27.847149 11556 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.933420 11522 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.958472 11522 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.959033 11522 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:27.979869 11555 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:27.987025 11565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:27.987455 11565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:27.996294 11565 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_5'
W20240503 02:12:28.288599 10161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 93423.7us
INSERT report
rows total: 2000
time total: 336.35 ms
time per row: 0.168175 ms
I20240503 02:12:28.493757 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:28.504307 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 8808b44fbad7430b99d4003405af2bec (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.504674 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 8808b44fbad7430b99d4003405af2bec (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.505072 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8808b44fbad7430b99d4003405af2bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.505182 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8808b44fbad7430b99d4003405af2bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.505122 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 63e002f92dd64155bf425c9aaaa314ab (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.505470 10470 tablet_service.cc:1467] Processing CreateTablet for tablet b3a66c03cb7541b88142e0e93089d86c (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.505571 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e002f92dd64155bf425c9aaaa314ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.506564 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 63e002f92dd64155bf425c9aaaa314ab (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.507084 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e002f92dd64155bf425c9aaaa314ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.508263 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a66c03cb7541b88142e0e93089d86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.513244 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 63e002f92dd64155bf425c9aaaa314ab (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.513216 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 8808b44fbad7430b99d4003405af2bec (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.513831 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 63e002f92dd64155bf425c9aaaa314ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.514106 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8808b44fbad7430b99d4003405af2bec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.517539 10073 tablet_service.cc:1467] Processing CreateTablet for tablet b3a66c03cb7541b88142e0e93089d86c (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.517946 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a66c03cb7541b88142e0e93089d86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.521270 10205 tablet_service.cc:1467] Processing CreateTablet for tablet b3a66c03cb7541b88142e0e93089d86c (DEFAULT_TABLE table=rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:28.521728 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b3a66c03cb7541b88142e0e93089d86c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:28.524183 11633 tablet_bootstrap.cc:492] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:28.526279 11633 tablet_bootstrap.cc:654] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.531453 11635 tablet_bootstrap.cc:492] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:28.531605 11633 tablet_bootstrap.cc:492] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:28.531770 11632 tablet_bootstrap.cc:492] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:28.531819 11633 ts_tablet_manager.cc:1367] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.004s
I20240503 02:12:28.533041 11633 raft_consensus.cc:348] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.533427 11633 raft_consensus.cc:374] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.533457 11634 tablet_bootstrap.cc:492] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:28.533623 11633 raft_consensus.cc:729] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.534027 11635 tablet_bootstrap.cc:654] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.534889 11632 tablet_bootstrap.cc:654] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.533968 11633 consensus_queue.cc:260] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.536808 11633 ts_tablet_manager.cc:1398] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:28.537297 11633 tablet_bootstrap.cc:492] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:28.539821 11633 tablet_bootstrap.cc:654] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.541087 11634 tablet_bootstrap.cc:654] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.549322 11632 tablet_bootstrap.cc:492] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:28.549526 11632 ts_tablet_manager.cc:1367] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.003s
I20240503 02:12:28.549793 11633 tablet_bootstrap.cc:492] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:28.549945 11634 tablet_bootstrap.cc:492] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:28.549996 11633 ts_tablet_manager.cc:1367] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.007s
I20240503 02:12:28.550256 11634 ts_tablet_manager.cc:1367] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.000s
I20240503 02:12:28.550972 11635 tablet_bootstrap.cc:492] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:28.550863 11632 raft_consensus.cc:348] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.551234 11632 raft_consensus.cc:374] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.551294 11635 ts_tablet_manager.cc:1367] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.016s
I20240503 02:12:28.551355 11632 raft_consensus.cc:729] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.551625 11632 consensus_queue.cc:260] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.552251 11632 ts_tablet_manager.cc:1398] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:28.552132 11634 raft_consensus.cc:348] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.552345 11634 raft_consensus.cc:374] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.552472 11634 raft_consensus.cc:729] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.551923 11633 raft_consensus.cc:348] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.552742 11633 raft_consensus.cc:374] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.552821 11633 raft_consensus.cc:729] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.552711 11635 raft_consensus.cc:348] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.553041 11635 raft_consensus.cc:374] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.553040 11634 consensus_queue.cc:260] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.553200 11635 raft_consensus.cc:729] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.553594 11632 tablet_bootstrap.cc:492] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:28.553517 11635 consensus_queue.cc:260] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.554497 11633 consensus_queue.cc:260] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.555137 11633 ts_tablet_manager.cc:1398] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:28.555483 11633 tablet_bootstrap.cc:492] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:28.557159 11634 ts_tablet_manager.cc:1398] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:28.559114 11632 tablet_bootstrap.cc:654] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.559299 11635 ts_tablet_manager.cc:1398] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:12:28.560526 11633 tablet_bootstrap.cc:654] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.560726 11635 tablet_bootstrap.cc:492] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:28.562944 11635 tablet_bootstrap.cc:654] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.563792 11632 tablet_bootstrap.cc:492] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:28.564172 11632 ts_tablet_manager.cc:1367] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20240503 02:12:28.565174 11632 raft_consensus.cc:348] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.565454 11632 raft_consensus.cc:374] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.565515 11635 tablet_bootstrap.cc:492] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:28.565541 11632 raft_consensus.cc:729] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.565636 11635 ts_tablet_manager.cc:1367] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:12:28.565799 11632 consensus_queue.cc:260] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.566411 11632 ts_tablet_manager.cc:1398] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:28.566385 11635 raft_consensus.cc:348] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.566617 11635 raft_consensus.cc:374] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.566673 11635 raft_consensus.cc:729] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.566972 11635 consensus_queue.cc:260] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.567528 11635 ts_tablet_manager.cc:1398] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:28.567998 11633 tablet_bootstrap.cc:492] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:28.568181 11633 ts_tablet_manager.cc:1367] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20240503 02:12:28.568212 11635 tablet_bootstrap.cc:492] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:28.569408 11633 raft_consensus.cc:348] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.569648 11633 raft_consensus.cc:374] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.569761 11633 raft_consensus.cc:729] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.570152 11635 tablet_bootstrap.cc:654] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:28.570075 11633 consensus_queue.cc:260] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.570665 11633 ts_tablet_manager.cc:1398] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:28.573110 11635 tablet_bootstrap.cc:492] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:28.573323 11635 ts_tablet_manager.cc:1367] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:12:28.575166 11635 raft_consensus.cc:348] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.575497 11635 raft_consensus.cc:374] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:28.575663 11635 raft_consensus.cc:729] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:28.576030 11635 consensus_queue.cc:260] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.576813 11635 ts_tablet_manager.cc:1398] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:28.590827 11617 raft_consensus.cc:482] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.591037 11617 raft_consensus.cc:504] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.591827 11617 leader_election.cc:288] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:28.592075 11617 raft_consensus.cc:482] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.592263 11617 raft_consensus.cc:504] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.592566 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a66c03cb7541b88142e0e93089d86c" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:28.592656 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a66c03cb7541b88142e0e93089d86c" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:28.592877 11617 leader_election.cc:288] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:28.592919 10094 raft_consensus.cc:2454] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:28.592906 10489 raft_consensus.cc:2454] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:28.593406 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8808b44fbad7430b99d4003405af2bec" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:28.593528 10159 leader_election.cc:302] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:28.593534 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8808b44fbad7430b99d4003405af2bec" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:28.593652 10489 raft_consensus.cc:2454] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:28.593735 10094 raft_consensus.cc:2454] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:28.593858 11617 raft_consensus.cc:2781] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.593969 11617 raft_consensus.cc:482] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.594110 11617 raft_consensus.cc:3037] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.594228 10161 leader_election.cc:302] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:28.594509 11522 raft_consensus.cc:2781] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.594624 11522 raft_consensus.cc:482] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.594724 11522 raft_consensus.cc:3037] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.597079 11522 raft_consensus.cc:504] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.597142 11617 raft_consensus.cc:504] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.597657 11522 leader_election.cc:288] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:28.597716 11617 leader_election.cc:288] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:28.598160 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a66c03cb7541b88142e0e93089d86c" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:28.598251 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8808b44fbad7430b99d4003405af2bec" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:28.598253 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8808b44fbad7430b99d4003405af2bec" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:28.598433 10490 raft_consensus.cc:3037] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.598464 10094 raft_consensus.cc:3037] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.598969 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b3a66c03cb7541b88142e0e93089d86c" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:28.599172 10092 raft_consensus.cc:3037] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.600642 10490 raft_consensus.cc:2454] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:28.601722 10094 raft_consensus.cc:2454] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:28.602120 10159 leader_election.cc:302] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:28.602150 10092 raft_consensus.cc:2454] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:28.602398 10489 raft_consensus.cc:3037] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.602532 10161 leader_election.cc:302] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:28.602819 11522 raft_consensus.cc:2781] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:28.602964 11522 raft_consensus.cc:686] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:28.603051 11612 raft_consensus.cc:2781] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:28.603199 11612 raft_consensus.cc:686] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:28.603513 11522 consensus_queue.cc:237] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:28.603574 11612 consensus_queue.cc:237] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:28.605628 10489 raft_consensus.cc:2454] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:28.610599 9952 catalog_manager.cc:5484] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:28.611287 9952 catalog_manager.cc:5484] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:28.796072 11556 raft_consensus.cc:482] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:28.796379 11556 raft_consensus.cc:504] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.797313 11556 leader_election.cc:288] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:28.798022 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e002f92dd64155bf425c9aaaa314ab" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:28.798051 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e002f92dd64155bf425c9aaaa314ab" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:28.798287 10489 raft_consensus.cc:2454] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:28.798311 10359 raft_consensus.cc:2454] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:28.798823 10028 leader_election.cc:302] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:28.799161 11556 raft_consensus.cc:2781] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:28.799260 11556 raft_consensus.cc:482] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:28.799371 11556 raft_consensus.cc:3037] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.802290 11556 raft_consensus.cc:504] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.803069 11556 leader_election.cc:288] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:28.803455 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e002f92dd64155bf425c9aaaa314ab" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:28.803617 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "63e002f92dd64155bf425c9aaaa314ab" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:28.803786 10489 raft_consensus.cc:3037] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.803874 10359 raft_consensus.cc:3037] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:28.806075 10359 raft_consensus.cc:2454] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:28.806529 10028 leader_election.cc:302] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:28.806795 11556 raft_consensus.cc:2781] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:28.806929 10489 raft_consensus.cc:2454] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:28.807401 11556 raft_consensus.cc:686] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:28.807729 11556 consensus_queue.cc:237] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:28.811168 9952 catalog_manager.cc:5484] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:29.080150 11612 consensus_queue.cc:1035] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.087350 11617 consensus_queue.cc:1035] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.127547 11617 consensus_queue.cc:1035] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.133915 11522 consensus_queue.cc:1035] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.188186 11556 consensus_queue.cc:1035] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:29.195010 11556 consensus_queue.cc:1035] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:29.254436 11649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:29.254845 11649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:29.263343 11649 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_6'
INSERT report
rows total: 2000
time total: 243.214 ms
time per row: 0.121607 ms
I20240503 02:12:29.672401 9943 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:29.682957 10337 tablet_service.cc:1467] Processing CreateTablet for tablet da288f76df0a409eb01ee431d3b83b49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.683287 10206 tablet_service.cc:1467] Processing CreateTablet for tablet da288f76df0a409eb01ee431d3b83b49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.683300 10073 tablet_service.cc:1467] Processing CreateTablet for tablet da288f76df0a409eb01ee431d3b83b49 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.683601 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da288f76df0a409eb01ee431d3b83b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.683766 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da288f76df0a409eb01ee431d3b83b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.683791 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da288f76df0a409eb01ee431d3b83b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.686326 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 31f72bc850c74289aa3a59ecc3dcc287 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.686731 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f72bc850c74289aa3a59ecc3dcc287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.693027 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 5547b33324db46c886287b4ee5035e43 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.693595 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5547b33324db46c886287b4ee5035e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.697080 11715 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:29.697758 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 31f72bc850c74289aa3a59ecc3dcc287 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.698127 11713 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:29.698344 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f72bc850c74289aa3a59ecc3dcc287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.700050 11715 tablet_bootstrap.cc:654] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.700078 11713 tablet_bootstrap.cc:654] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.701020 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 5547b33324db46c886287b4ee5035e43 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.701390 11714 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:29.701579 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5547b33324db46c886287b4ee5035e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.703488 11714 tablet_bootstrap.cc:654] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.705102 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 31f72bc850c74289aa3a59ecc3dcc287 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.705633 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f72bc850c74289aa3a59ecc3dcc287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.707074 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 5547b33324db46c886287b4ee5035e43 (DEFAULT_TABLE table=rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:29.707679 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5547b33324db46c886287b4ee5035e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:29.711691 11715 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:29.711963 11715 ts_tablet_manager.cc:1367] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.003s
I20240503 02:12:29.713176 11715 raft_consensus.cc:348] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:29.713647 11715 raft_consensus.cc:374] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.713801 11715 raft_consensus.cc:729] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.714265 11715 consensus_queue.cc:260] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:29.715036 11715 ts_tablet_manager.cc:1398] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:29.715574 11715 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:29.718143 11715 tablet_bootstrap.cc:654] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.721165 11714 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:29.721482 11714 ts_tablet_manager.cc:1367] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.005s
I20240503 02:12:29.722797 11714 raft_consensus.cc:348] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:29.723240 11714 raft_consensus.cc:374] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.723423 11714 raft_consensus.cc:729] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.723881 11714 consensus_queue.cc:260] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:29.724870 11714 ts_tablet_manager.cc:1398] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:29.727517 11715 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:29.727787 11715 ts_tablet_manager.cc:1367] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 02:12:29.728772 11715 raft_consensus.cc:348] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.729135 11715 raft_consensus.cc:374] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.729255 11715 raft_consensus.cc:729] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.729578 11715 consensus_queue.cc:260] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.730306 11715 ts_tablet_manager.cc:1398] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:29.741034 11713 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:29.741307 11713 ts_tablet_manager.cc:1367] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.043s user 0.018s sys 0.022s
I20240503 02:12:29.742311 11713 raft_consensus.cc:348] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:29.742666 11713 raft_consensus.cc:374] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.742842 11713 raft_consensus.cc:729] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.743144 11713 consensus_queue.cc:260] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:29.744899 11713 ts_tablet_manager.cc:1398] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:29.745244 11713 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:29.747503 11714 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:29.747732 11713 tablet_bootstrap.cc:654] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.750353 11714 tablet_bootstrap.cc:654] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.755228 11720 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:29.755543 11713 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:29.755802 11713 ts_tablet_manager.cc:1367] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.002s
I20240503 02:12:29.756729 11713 raft_consensus.cc:348] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:29.757052 11713 raft_consensus.cc:374] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.757181 11713 raft_consensus.cc:729] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.757539 11713 consensus_queue.cc:260] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:29.761107 11713 ts_tablet_manager.cc:1398] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:29.761940 11720 tablet_bootstrap.cc:654] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.764542 11713 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:29.767306 11713 tablet_bootstrap.cc:654] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.769811 11714 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:29.770013 11714 ts_tablet_manager.cc:1367] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.010s
I20240503 02:12:29.770308 11713 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:29.770447 11713 ts_tablet_manager.cc:1367] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.001s
I20240503 02:12:29.770874 11714 raft_consensus.cc:348] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:29.771128 11714 raft_consensus.cc:374] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.771209 11714 raft_consensus.cc:729] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.771288 11713 raft_consensus.cc:348] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.771505 11713 raft_consensus.cc:374] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.771461 11714 consensus_queue.cc:260] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:29.771613 11713 raft_consensus.cc:729] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.771888 11713 consensus_queue.cc:260] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.772068 11714 ts_tablet_manager.cc:1398] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:29.773684 11713 ts_tablet_manager.cc:1398] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:29.773936 11720 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:29.774158 11720 ts_tablet_manager.cc:1367] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.007s
I20240503 02:12:29.775154 11720 raft_consensus.cc:348] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:29.775543 11720 raft_consensus.cc:374] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.775667 11720 raft_consensus.cc:729] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.776006 11720 consensus_queue.cc:260] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:29.776686 11720 ts_tablet_manager.cc:1398] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:29.777120 11720 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:29.779204 11720 tablet_bootstrap.cc:654] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:29.797188 11720 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:29.797580 11720 ts_tablet_manager.cc:1367] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.015s
I20240503 02:12:29.798421 11720 raft_consensus.cc:348] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.798636 11720 raft_consensus.cc:374] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:29.798723 11720 raft_consensus.cc:729] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:29.799042 11720 consensus_queue.cc:260] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.799571 11720 ts_tablet_manager.cc:1398] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:29.857120 11556 raft_consensus.cc:482] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:29.857374 11556 raft_consensus.cc:504] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.858345 11556 leader_election.cc:288] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:29.859031 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5547b33324db46c886287b4ee5035e43" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:29.859036 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5547b33324db46c886287b4ee5035e43" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:29.859297 10490 raft_consensus.cc:2454] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:29.859341 10358 raft_consensus.cc:2454] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:29.859889 10026 leader_election.cc:302] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:29.860214 11556 raft_consensus.cc:2781] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:29.860352 11556 raft_consensus.cc:482] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:29.860484 11556 raft_consensus.cc:3037] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:29.862860 11556 raft_consensus.cc:504] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.863595 11556 leader_election.cc:288] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:29.864238 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5547b33324db46c886287b4ee5035e43" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:29.864241 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5547b33324db46c886287b4ee5035e43" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:29.864512 10358 raft_consensus.cc:3037] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:29.864523 10490 raft_consensus.cc:3037] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:29.867764 10358 raft_consensus.cc:2454] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:29.867753 10490 raft_consensus.cc:2454] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:29.868309 10028 leader_election.cc:302] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:29.868726 11556 raft_consensus.cc:2781] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:29.868870 11556 raft_consensus.cc:686] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:29.869452 11556 consensus_queue.cc:237] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:29.873030 9949 catalog_manager.cc:5484] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:30.000042 11346 raft_consensus.cc:482] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:30.000319 11346 raft_consensus.cc:504] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:30.001008 11346 leader_election.cc:288] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:30.001483 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f72bc850c74289aa3a59ecc3dcc287" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:30.001572 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f72bc850c74289aa3a59ecc3dcc287" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:30.001709 10227 raft_consensus.cc:2454] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:30.001870 10358 raft_consensus.cc:2454] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:30.002245 10426 leader_election.cc:302] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:30.002553 11346 raft_consensus.cc:2781] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:30.002667 11346 raft_consensus.cc:482] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:30.002808 11346 raft_consensus.cc:3037] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.004974 11346 raft_consensus.cc:504] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:30.005659 11346 leader_election.cc:288] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:30.006197 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f72bc850c74289aa3a59ecc3dcc287" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:30.006273 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "31f72bc850c74289aa3a59ecc3dcc287" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:30.006445 10227 raft_consensus.cc:3037] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.006546 10358 raft_consensus.cc:3037] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.009680 10358 raft_consensus.cc:2454] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:30.009696 10227 raft_consensus.cc:2454] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:30.010129 10426 leader_election.cc:302] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:30.010434 11346 raft_consensus.cc:2781] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:30.010905 11346 raft_consensus.cc:686] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:30.011247 11346 consensus_queue.cc:237] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:30.014601 9949 catalog_manager.cc:5484] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:30.127018 11494 raft_consensus.cc:482] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:30.127259 11494 raft_consensus.cc:504] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:30.128006 11494 leader_election.cc:288] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:30.128818 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da288f76df0a409eb01ee431d3b83b49" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:30.128947 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da288f76df0a409eb01ee431d3b83b49" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:30.129132 10094 raft_consensus.cc:2454] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:30.129263 10227 raft_consensus.cc:2454] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:30.129670 10294 leader_election.cc:302] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:30.129998 11494 raft_consensus.cc:2781] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:30.130148 11494 raft_consensus.cc:482] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:30.130254 11494 raft_consensus.cc:3037] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.132849 11494 raft_consensus.cc:504] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:30.133653 11494 leader_election.cc:288] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:30.134065 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da288f76df0a409eb01ee431d3b83b49" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:30.134270 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da288f76df0a409eb01ee431d3b83b49" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:30.134371 10094 raft_consensus.cc:3037] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.134557 10227 raft_consensus.cc:3037] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:30.136607 10094 raft_consensus.cc:2454] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:30.136864 10227 raft_consensus.cc:2454] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:30.137044 10294 leader_election.cc:302] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:30.137353 11494 raft_consensus.cc:2781] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:30.137533 11494 raft_consensus.cc:686] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:30.137861 11494 consensus_queue.cc:237] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:30.141311 9949 catalog_manager.cc:5484] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:30.282310 11556 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:30.288312 11556 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:30.415757 11346 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:30.423274 11727 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:30.561419 11494 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:30.567554 11494 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:30.638734 11731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:30.639127 11731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:30.647822 11731 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_7'
INSERT report
rows total: 2000
time total: 349.465 ms
time per row: 0.174732 ms
-----------------------------------------------------
Suppressions used:
count bytes template
2 596 libcrypto.so.1.1
-----------------------------------------------------
I20240503 02:12:32.117650 9943 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:32.129002 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1681e90d92423a8fd4f5702cff768f (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.129281 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1681e90d92423a8fd4f5702cff768f (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.129623 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1681e90d92423a8fd4f5702cff768f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.129683 10206 tablet_service.cc:1467] Processing CreateTablet for tablet d0aabb6d297143e2a7bf935fd4a01e84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.129804 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1681e90d92423a8fd4f5702cff768f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.129859 10074 tablet_service.cc:1467] Processing CreateTablet for tablet d0aabb6d297143e2a7bf935fd4a01e84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.130071 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aabb6d297143e2a7bf935fd4a01e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.130316 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aabb6d297143e2a7bf935fd4a01e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.130542 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 2a1681e90d92423a8fd4f5702cff768f (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.131045 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2a1681e90d92423a8fd4f5702cff768f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.133081 10471 tablet_service.cc:1467] Processing CreateTablet for tablet d0aabb6d297143e2a7bf935fd4a01e84 (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.133539 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d0aabb6d297143e2a7bf935fd4a01e84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.133553 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 8d10cf9defbc44978489aa90f16db80b (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.133935 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d10cf9defbc44978489aa90f16db80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.137598 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 8d10cf9defbc44978489aa90f16db80b (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.137997 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d10cf9defbc44978489aa90f16db80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.144941 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 8d10cf9defbc44978489aa90f16db80b (DEFAULT_TABLE table=rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:32.145504 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8d10cf9defbc44978489aa90f16db80b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:32.148017 11811 tablet_bootstrap.cc:492] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:32.150164 11811 tablet_bootstrap.cc:654] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.151641 11809 tablet_bootstrap.cc:492] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:32.151877 11812 tablet_bootstrap.cc:492] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:32.151875 11810 tablet_bootstrap.cc:492] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:32.153887 11812 tablet_bootstrap.cc:654] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.154199 11809 tablet_bootstrap.cc:654] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.154362 11810 tablet_bootstrap.cc:654] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.158250 11811 tablet_bootstrap.cc:492] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:32.158499 11811 ts_tablet_manager.cc:1367] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 02:12:32.159483 11811 raft_consensus.cc:348] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.159731 11811 raft_consensus.cc:374] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.159835 11811 raft_consensus.cc:729] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.160125 11809 tablet_bootstrap.cc:492] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:32.160070 11811 consensus_queue.cc:260] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.160319 11809 ts_tablet_manager.cc:1367] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20240503 02:12:32.160791 11811 ts_tablet_manager.cc:1398] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:32.161463 11809 raft_consensus.cc:348] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.161798 11809 raft_consensus.cc:374] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.161938 11809 raft_consensus.cc:729] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.162272 11809 consensus_queue.cc:260] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.162791 11811 tablet_bootstrap.cc:492] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:32.164024 11810 tablet_bootstrap.cc:492] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:32.164178 11810 ts_tablet_manager.cc:1367] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20240503 02:12:32.165601 11811 tablet_bootstrap.cc:654] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.167289 11809 ts_tablet_manager.cc:1398] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:32.167543 11810 raft_consensus.cc:348] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.167953 11810 raft_consensus.cc:374] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.168131 11810 raft_consensus.cc:729] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.168550 11810 consensus_queue.cc:260] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.171490 11809 tablet_bootstrap.cc:492] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:32.173389 11810 ts_tablet_manager.cc:1398] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:12:32.173993 11809 tablet_bootstrap.cc:654] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.176651 11812 tablet_bootstrap.cc:492] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:32.176848 11812 ts_tablet_manager.cc:1367] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.012s
I20240503 02:12:32.177213 11809 tablet_bootstrap.cc:492] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:32.177111 11811 tablet_bootstrap.cc:492] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:32.177383 11809 ts_tablet_manager.cc:1367] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:12:32.177414 11811 ts_tablet_manager.cc:1367] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20240503 02:12:32.177628 11810 tablet_bootstrap.cc:492] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:32.177788 11812 raft_consensus.cc:348] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.178110 11812 raft_consensus.cc:374] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.178277 11812 raft_consensus.cc:729] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.178184 11809 raft_consensus.cc:348] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.178267 11811 raft_consensus.cc:348] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.178447 11809 raft_consensus.cc:374] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.178587 11809 raft_consensus.cc:729] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.178612 11811 raft_consensus.cc:374] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.178647 11812 consensus_queue.cc:260] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.178742 11811 raft_consensus.cc:729] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.178897 11809 consensus_queue.cc:260] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.179232 11811 consensus_queue.cc:260] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.179715 11810 tablet_bootstrap.cc:654] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.180577 11809 ts_tablet_manager.cc:1398] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:32.180598 11812 ts_tablet_manager.cc:1398] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:32.180861 11812 tablet_bootstrap.cc:492] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:32.180877 11809 tablet_bootstrap.cc:492] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:32.181461 11811 ts_tablet_manager.cc:1398] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:32.182495 11809 tablet_bootstrap.cc:654] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.182510 11812 tablet_bootstrap.cc:654] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:32.190305 11809 tablet_bootstrap.cc:492] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:32.190506 11809 ts_tablet_manager.cc:1367] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:12:32.191059 11810 tablet_bootstrap.cc:492] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:32.191206 11812 tablet_bootstrap.cc:492] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:32.191263 11810 ts_tablet_manager.cc:1367] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.003s
I20240503 02:12:32.191407 11812 ts_tablet_manager.cc:1367] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.001s
I20240503 02:12:32.191458 11809 raft_consensus.cc:348] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.191717 11809 raft_consensus.cc:374] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.191812 11809 raft_consensus.cc:729] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.192067 11810 raft_consensus.cc:348] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.192075 11809 consensus_queue.cc:260] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.192281 11810 raft_consensus.cc:374] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.192369 11810 raft_consensus.cc:729] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.192270 11812 raft_consensus.cc:348] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.192703 11809 ts_tablet_manager.cc:1398] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:32.192716 11812 raft_consensus.cc:374] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:32.192795 11812 raft_consensus.cc:729] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:32.192700 11810 consensus_queue.cc:260] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.193065 11812 consensus_queue.cc:260] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.193410 11810 ts_tablet_manager.cc:1398] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:32.193635 11812 ts_tablet_manager.cc:1398] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:32.251709 11803 raft_consensus.cc:482] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.251950 11803 raft_consensus.cc:504] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.252713 11803 leader_election.cc:288] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:32.253173 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aabb6d297143e2a7bf935fd4a01e84" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:32.253196 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aabb6d297143e2a7bf935fd4a01e84" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:32.253492 10226 raft_consensus.cc:2454] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:32.253487 10490 raft_consensus.cc:2454] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:32.253994 10026 leader_election.cc:302] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:32.254281 11803 raft_consensus.cc:2781] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:32.254413 11803 raft_consensus.cc:482] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:32.254509 11803 raft_consensus.cc:3037] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.256697 11803 raft_consensus.cc:504] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.257336 11803 leader_election.cc:288] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:32.257830 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aabb6d297143e2a7bf935fd4a01e84" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:32.257881 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d0aabb6d297143e2a7bf935fd4a01e84" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:32.258075 10226 raft_consensus.cc:3037] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.258157 10490 raft_consensus.cc:3037] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.260211 10226 raft_consensus.cc:2454] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:32.260650 10028 leader_election.cc:302] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:32.260929 11803 raft_consensus.cc:2781] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.261094 11803 raft_consensus.cc:686] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:32.261174 10490 raft_consensus.cc:2454] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:32.261425 11803 consensus_queue.cc:237] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:32.265200 9950 catalog_manager.cc:5484] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:32.365585 11781 raft_consensus.cc:482] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.365844 11781 raft_consensus.cc:504] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.366756 11781 leader_election.cc:288] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:32.367141 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d10cf9defbc44978489aa90f16db80b" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:32.367440 10358 raft_consensus.cc:2454] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:32.367420 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d10cf9defbc44978489aa90f16db80b" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:32.367677 10490 raft_consensus.cc:2454] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:32.367985 10028 leader_election.cc:302] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:32.368290 11781 raft_consensus.cc:2781] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:32.368393 11781 raft_consensus.cc:482] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:32.368497 11781 raft_consensus.cc:3037] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.370641 11781 raft_consensus.cc:504] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.371199 11781 leader_election.cc:288] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:32.371685 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d10cf9defbc44978489aa90f16db80b" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:32.371685 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8d10cf9defbc44978489aa90f16db80b" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:32.371915 10490 raft_consensus.cc:3037] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.371915 10358 raft_consensus.cc:3037] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.375005 10358 raft_consensus.cc:2454] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:32.375005 10490 raft_consensus.cc:2454] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:32.375520 10026 leader_election.cc:302] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:32.375869 11781 raft_consensus.cc:2781] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.376014 11781 raft_consensus.cc:686] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:32.376339 11781 consensus_queue.cc:237] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.379735 9950 catalog_manager.cc:5484] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:32.496305 11781 raft_consensus.cc:482] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:32.496608 11781 raft_consensus.cc:504] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.497265 11781 leader_election.cc:288] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:32.497820 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1681e90d92423a8fd4f5702cff768f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:32.497897 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1681e90d92423a8fd4f5702cff768f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:32.498149 10226 raft_consensus.cc:2454] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:32.498157 10358 raft_consensus.cc:2454] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:32.498574 10028 leader_election.cc:302] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:32.498946 11803 raft_consensus.cc:2781] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:32.499099 11803 raft_consensus.cc:482] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:32.499208 11803 raft_consensus.cc:3037] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.501828 11803 raft_consensus.cc:504] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.502480 11803 leader_election.cc:288] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:32.502964 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1681e90d92423a8fd4f5702cff768f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:32.503067 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2a1681e90d92423a8fd4f5702cff768f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:32.503154 10226 raft_consensus.cc:3037] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.503314 10358 raft_consensus.cc:3037] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:32.505249 10226 raft_consensus.cc:2454] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:32.505435 10358 raft_consensus.cc:2454] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:32.505679 10028 leader_election.cc:302] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:32.505952 11803 raft_consensus.cc:2781] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:32.506090 11803 raft_consensus.cc:686] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:32.506390 11803 consensus_queue.cc:237] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:32.510915 9950 catalog_manager.cc:5484] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:32.749115 11803 consensus_queue.cc:1035] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.757445 11803 consensus_queue.cc:1035] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.868808 11829 consensus_queue.cc:1035] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.875202 11781 consensus_queue.cc:1035] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.883836 11829 consensus_queue.cc:1035] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:32.889325 11829 consensus_queue.cc:1035] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:32.918004 11825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:32.918417 11825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:32.926303 11825 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_8'
INSERT report
rows total: 2000
time total: 208.191 ms
time per row: 0.104096 ms
I20240503 02:12:33.303682 9944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:33.313980 10205 tablet_service.cc:1467] Processing CreateTablet for tablet af7beeafa08f477796e5070fbc177dcc (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.314133 10469 tablet_service.cc:1467] Processing CreateTablet for tablet af7beeafa08f477796e5070fbc177dcc (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.314613 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7beeafa08f477796e5070fbc177dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.314639 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7beeafa08f477796e5070fbc177dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.314960 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 206e6a3850ff4242b30fa10c3f751fd2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.315356 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 206e6a3850ff4242b30fa10c3f751fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.317430 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 206e6a3850ff4242b30fa10c3f751fd2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.317900 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 206e6a3850ff4242b30fa10c3f751fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.321439 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc2733b737b431684ce439c3d3fa762 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.321501 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc2733b737b431684ce439c3d3fa762 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.321863 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc2733b737b431684ce439c3d3fa762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.322027 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc2733b737b431684ce439c3d3fa762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.324913 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 206e6a3850ff4242b30fa10c3f751fd2 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.325057 10338 tablet_service.cc:1467] Processing CreateTablet for tablet af7beeafa08f477796e5070fbc177dcc (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.325325 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 206e6a3850ff4242b30fa10c3f751fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.325558 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7beeafa08f477796e5070fbc177dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.328716 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 0dc2733b737b431684ce439c3d3fa762 (DEFAULT_TABLE table=rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:33.329337 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc2733b737b431684ce439c3d3fa762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:33.338661 11893 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:33.344542 11893 tablet_bootstrap.cc:654] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.349296 11894 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:33.350471 11895 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:33.350952 11893 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:33.351109 11893 ts_tablet_manager.cc:1367] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20240503 02:12:33.351943 11896 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:33.352092 11893 raft_consensus.cc:348] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.352265 11894 tablet_bootstrap.cc:654] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.352339 11893 raft_consensus.cc:374] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.352428 11893 raft_consensus.cc:729] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.352717 11893 consensus_queue.cc:260] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.354305 11895 tablet_bootstrap.cc:654] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.355125 11893 ts_tablet_manager.cc:1398] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:12:33.355410 11893 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:33.357188 11893 tablet_bootstrap.cc:654] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.362085 11894 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:33.362282 11894 ts_tablet_manager.cc:1367] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20240503 02:12:33.363466 11894 raft_consensus.cc:348] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.363790 11894 raft_consensus.cc:374] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.363907 11894 raft_consensus.cc:729] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.364343 11894 consensus_queue.cc:260] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.365226 11894 ts_tablet_manager.cc:1398] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20240503 02:12:33.366739 11895 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:33.366935 11893 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:33.366964 11895 ts_tablet_manager.cc:1367] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.017s user 0.010s sys 0.003s
I20240503 02:12:33.367120 11893 ts_tablet_manager.cc:1367] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.008s
I20240503 02:12:33.367444 11896 tablet_bootstrap.cc:654] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.368036 11895 raft_consensus.cc:348] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.368060 11893 raft_consensus.cc:348] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.368333 11895 raft_consensus.cc:374] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.368347 11893 raft_consensus.cc:374] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.368415 11893 raft_consensus.cc:729] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.368515 11895 raft_consensus.cc:729] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.368717 11893 consensus_queue.cc:260] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.368799 11895 consensus_queue.cc:260] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.369621 11895 ts_tablet_manager.cc:1398] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:33.369997 11895 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:33.372220 11895 tablet_bootstrap.cc:654] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.385118 11893 ts_tablet_manager.cc:1398] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.018s user 0.007s sys 0.009s
I20240503 02:12:33.385648 11893 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:33.386178 11895 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:33.386337 11896 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:33.386585 11896 ts_tablet_manager.cc:1367] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.000s
I20240503 02:12:33.386580 11895 ts_tablet_manager.cc:1367] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.006s
I20240503 02:12:33.387533 11895 raft_consensus.cc:348] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.387804 11895 raft_consensus.cc:374] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.387898 11895 raft_consensus.cc:729] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.388067 11893 tablet_bootstrap.cc:654] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.388288 11896 raft_consensus.cc:348] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.388314 11895 consensus_queue.cc:260] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.388656 11896 raft_consensus.cc:374] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.389351 11895 ts_tablet_manager.cc:1398] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:33.389478 11896 raft_consensus.cc:729] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.389770 11895 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:33.389755 11896 consensus_queue.cc:260] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.390697 11893 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:33.390856 11893 ts_tablet_manager.cc:1367] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:12:33.391760 11895 tablet_bootstrap.cc:654] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.391739 11893 raft_consensus.cc:348] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.391973 11893 raft_consensus.cc:374] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.392076 11893 raft_consensus.cc:729] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.392350 11893 consensus_queue.cc:260] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.393147 11893 ts_tablet_manager.cc:1398] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:33.393184 11896 ts_tablet_manager.cc:1398] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:33.393610 11896 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:33.395783 11896 tablet_bootstrap.cc:654] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:33.401350 11895 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:33.401736 11895 ts_tablet_manager.cc:1367] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20240503 02:12:33.402484 11895 raft_consensus.cc:348] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.402686 11895 raft_consensus.cc:374] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.402802 11895 raft_consensus.cc:729] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.403048 11895 consensus_queue.cc:260] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.403621 11895 ts_tablet_manager.cc:1398] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:33.404891 11896 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:33.405055 11896 ts_tablet_manager.cc:1367] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 02:12:33.405768 11896 raft_consensus.cc:348] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.406024 11896 raft_consensus.cc:374] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:33.406117 11896 raft_consensus.cc:729] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:33.406369 11896 consensus_queue.cc:260] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.406913 11896 ts_tablet_manager.cc:1398] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:33.427928 11902 raft_consensus.cc:482] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.428194 11902 raft_consensus.cc:504] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.429085 11902 leader_election.cc:288] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:33.429682 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7beeafa08f477796e5070fbc177dcc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:33.429764 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7beeafa08f477796e5070fbc177dcc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:33.429998 10490 raft_consensus.cc:2454] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:33.430071 10359 raft_consensus.cc:2454] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:33.430555 10159 leader_election.cc:302] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:33.430927 11902 raft_consensus.cc:2781] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.431083 11902 raft_consensus.cc:482] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.431188 11902 raft_consensus.cc:3037] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.433975 11902 raft_consensus.cc:504] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.434651 11902 leader_election.cc:288] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:33.435160 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7beeafa08f477796e5070fbc177dcc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:33.435247 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af7beeafa08f477796e5070fbc177dcc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:33.435416 10490 raft_consensus.cc:3037] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.435508 10359 raft_consensus.cc:3037] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.438267 10490 raft_consensus.cc:2454] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:33.438275 10359 raft_consensus.cc:2454] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:33.438702 10159 leader_election.cc:302] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:33.439003 11902 raft_consensus.cc:2781] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.439131 11902 raft_consensus.cc:686] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:33.439476 11902 consensus_queue.cc:237] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.443140 9949 catalog_manager.cc:5484] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:33.518965 11799 raft_consensus.cc:482] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.519214 11799 raft_consensus.cc:504] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.520089 11799 leader_election.cc:288] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:33.520908 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206e6a3850ff4242b30fa10c3f751fd2" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:33.520972 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206e6a3850ff4242b30fa10c3f751fd2" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:33.521239 10094 raft_consensus.cc:2454] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:33.521262 10226 raft_consensus.cc:2454] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:33.521750 10294 leader_election.cc:302] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:33.522047 11799 raft_consensus.cc:2781] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.522186 11799 raft_consensus.cc:482] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.522289 11799 raft_consensus.cc:3037] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.524626 11799 raft_consensus.cc:504] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.525343 11799 leader_election.cc:288] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:33.526144 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206e6a3850ff4242b30fa10c3f751fd2" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:33.526154 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206e6a3850ff4242b30fa10c3f751fd2" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:33.526423 10094 raft_consensus.cc:3037] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.526450 10226 raft_consensus.cc:3037] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.529846 10226 raft_consensus.cc:2454] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:33.530934 10294 leader_election.cc:302] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:33.531327 10094 raft_consensus.cc:2454] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:33.531369 11799 raft_consensus.cc:2781] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.532274 11799 raft_consensus.cc:686] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:33.532644 11799 consensus_queue.cc:237] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.536615 9949 catalog_manager.cc:5484] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:33.593086 11902 raft_consensus.cc:482] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:33.593349 11902 raft_consensus.cc:504] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.594084 11902 leader_election.cc:288] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:33.594791 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc2733b737b431684ce439c3d3fa762" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:33.594791 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc2733b737b431684ce439c3d3fa762" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:33.595064 10490 raft_consensus.cc:2454] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:33.595079 10359 raft_consensus.cc:2454] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:33.595535 10161 leader_election.cc:302] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:33.595911 11902 raft_consensus.cc:2781] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:33.596032 11902 raft_consensus.cc:482] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:33.596153 11902 raft_consensus.cc:3037] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.598440 11902 raft_consensus.cc:504] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.599069 11902 leader_election.cc:288] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:33.599573 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc2733b737b431684ce439c3d3fa762" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:33.599573 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0dc2733b737b431684ce439c3d3fa762" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:33.599820 10359 raft_consensus.cc:3037] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.599820 10490 raft_consensus.cc:3037] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:33.602905 10359 raft_consensus.cc:2454] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:33.602923 10490 raft_consensus.cc:2454] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:33.603349 10159 leader_election.cc:302] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:33.603655 11902 raft_consensus.cc:2781] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:33.603826 11902 raft_consensus.cc:686] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:33.604146 11902 consensus_queue.cc:237] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:33.607671 9949 catalog_manager.cc:5484] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:33.852046 11902 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:33.862943 11902 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.004874 11799 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.013520 11799 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.023443 11902 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:34.030978 11802 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:34.075011 11911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:34.075481 11911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:34.084614 11911 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_9'
W20240503 02:12:34.349843 10161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 73004.2us
INSERT report
rows total: 2000
time total: 269.531 ms
time per row: 0.134766 ms
I20240503 02:12:34.520213 9944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:34.530134 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 92c8ac796c0441199e0a47a8e4d3e1b0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.530658 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.530771 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c8ac796c0441199e0a47a8e4d3e1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.531102 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.532163 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.532614 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.532847 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 94d6673d883d4e788ad3e7fe020d7713 (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.533123 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.533210 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d6673d883d4e788ad3e7fe020d7713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.536633 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 92c8ac796c0441199e0a47a8e4d3e1b0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.537009 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c8ac796c0441199e0a47a8e4d3e1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.536929 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 92c8ac796c0441199e0a47a8e4d3e1b0 (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.537392 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 92c8ac796c0441199e0a47a8e4d3e1b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.541275 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 94d6673d883d4e788ad3e7fe020d7713 (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.541409 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.541652 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d6673d883d4e788ad3e7fe020d7713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.545580 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 94d6673d883d4e788ad3e7fe020d7713 (DEFAULT_TABLE table=rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:34.546042 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 94d6673d883d4e788ad3e7fe020d7713. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:34.546217 11979 tablet_bootstrap.cc:492] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:34.550834 11979 tablet_bootstrap.cc:654] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.554682 11981 tablet_bootstrap.cc:492] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:34.556061 11980 tablet_bootstrap.cc:492] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:34.556576 11981 tablet_bootstrap.cc:654] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.557979 11980 tablet_bootstrap.cc:654] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.559814 11982 tablet_bootstrap.cc:492] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:34.560693 11980 tablet_bootstrap.cc:492] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:34.560851 11980 ts_tablet_manager.cc:1367] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20240503 02:12:34.561707 11980 raft_consensus.cc:348] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.561960 11980 raft_consensus.cc:374] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.562048 11980 raft_consensus.cc:729] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.562268 11980 consensus_queue.cc:260] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.562935 11980 ts_tablet_manager.cc:1398] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:34.563079 11982 tablet_bootstrap.cc:654] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.563342 11980 tablet_bootstrap.cc:492] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:34.564795 11979 tablet_bootstrap.cc:492] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:34.565012 11979 ts_tablet_manager.cc:1367] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20240503 02:12:34.565739 11980 tablet_bootstrap.cc:654] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.565903 11979 raft_consensus.cc:348] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.566082 11979 raft_consensus.cc:374] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.566171 11979 raft_consensus.cc:729] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.566426 11979 consensus_queue.cc:260] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.566736 11982 tablet_bootstrap.cc:492] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:34.566886 11982 ts_tablet_manager.cc:1367] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20240503 02:12:34.567677 11982 raft_consensus.cc:348] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.568768 11982 raft_consensus.cc:374] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.568810 11979 ts_tablet_manager.cc:1398] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:34.568926 11982 raft_consensus.cc:729] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.569345 11981 tablet_bootstrap.cc:492] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:34.569228 11982 consensus_queue.cc:260] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.569528 11981 ts_tablet_manager.cc:1367] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.005s
I20240503 02:12:34.569907 11982 ts_tablet_manager.cc:1398] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:34.570399 11981 raft_consensus.cc:348] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.570649 11981 raft_consensus.cc:374] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.570792 11981 raft_consensus.cc:729] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.571082 11981 consensus_queue.cc:260] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.573503 11981 ts_tablet_manager.cc:1398] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:34.573596 11980 tablet_bootstrap.cc:492] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:34.573804 11980 ts_tablet_manager.cc:1367] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.000s
I20240503 02:12:34.573827 11981 tablet_bootstrap.cc:492] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:34.574548 11980 raft_consensus.cc:348] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.574776 11980 raft_consensus.cc:374] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.574848 11980 raft_consensus.cc:729] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.575062 11980 consensus_queue.cc:260] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.575644 11981 tablet_bootstrap.cc:654] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.575722 11980 ts_tablet_manager.cc:1398] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:34.575992 11980 tablet_bootstrap.cc:492] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:34.577620 11980 tablet_bootstrap.cc:654] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.578375 11982 tablet_bootstrap.cc:492] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:34.597553 11982 tablet_bootstrap.cc:654] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.599323 11980 tablet_bootstrap.cc:492] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:34.599581 11980 ts_tablet_manager.cc:1367] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.024s user 0.014s sys 0.007s
I20240503 02:12:34.600363 11980 raft_consensus.cc:348] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.600690 11980 raft_consensus.cc:374] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.600823 11980 raft_consensus.cc:729] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.601066 11980 consensus_queue.cc:260] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.601655 11980 ts_tablet_manager.cc:1398] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:34.606655 11981 tablet_bootstrap.cc:492] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:34.606905 11981 ts_tablet_manager.cc:1367] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.033s user 0.011s sys 0.014s
I20240503 02:12:34.607698 11982 tablet_bootstrap.cc:492] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:34.607901 11982 ts_tablet_manager.cc:1367] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.007s
I20240503 02:12:34.607812 11981 raft_consensus.cc:348] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.608074 11981 raft_consensus.cc:374] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.608177 11981 raft_consensus.cc:729] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.608477 11981 consensus_queue.cc:260] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.608784 11982 raft_consensus.cc:348] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.609053 11982 raft_consensus.cc:374] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.609166 11982 raft_consensus.cc:729] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.609198 11981 ts_tablet_manager.cc:1398] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:34.609535 11981 tablet_bootstrap.cc:492] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:34.609472 11982 consensus_queue.cc:260] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.611507 11982 ts_tablet_manager.cc:1398] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:34.612811 11981 tablet_bootstrap.cc:654] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:34.615018 11981 tablet_bootstrap.cc:492] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:34.615195 11981 ts_tablet_manager.cc:1367] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:12:34.616034 11981 raft_consensus.cc:348] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.616194 11981 raft_consensus.cc:374] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:34.616266 11981 raft_consensus.cc:729] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:34.616436 11981 consensus_queue.cc:260] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.617048 11981 ts_tablet_manager.cc:1398] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:34.654311 11965 raft_consensus.cc:482] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.654544 11965 raft_consensus.cc:504] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.655277 11965 leader_election.cc:288] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:34.655898 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c8ac796c0441199e0a47a8e4d3e1b0" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:34.655920 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c8ac796c0441199e0a47a8e4d3e1b0" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:34.656169 10092 raft_consensus.cc:2454] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:34.656181 10490 raft_consensus.cc:2454] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:34.656693 10161 leader_election.cc:302] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:34.656992 11965 raft_consensus.cc:2781] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.657096 11965 raft_consensus.cc:482] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.657187 11965 raft_consensus.cc:3037] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.659256 11965 raft_consensus.cc:504] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.659823 11965 leader_election.cc:288] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:34.660284 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c8ac796c0441199e0a47a8e4d3e1b0" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:34.660334 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "92c8ac796c0441199e0a47a8e4d3e1b0" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:34.660548 10092 raft_consensus.cc:3037] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.660586 10490 raft_consensus.cc:3037] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.662127 11908 raft_consensus.cc:482] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.662320 11908 raft_consensus.cc:504] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.663100 10092 raft_consensus.cc:2454] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:34.663200 11908 leader_election.cc:288] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:34.663338 10490 raft_consensus.cc:2454] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:34.663815 10161 leader_election.cc:302] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:34.664124 11965 raft_consensus.cc:2781] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.664108 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdc2a02fa0d42eb93bbbe6213b4723a" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:34.664116 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdc2a02fa0d42eb93bbbe6213b4723a" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:34.664336 10490 raft_consensus.cc:2454] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:34.664332 10092 raft_consensus.cc:2454] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:34.664393 11965 raft_consensus.cc:686] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:34.665056 10292 leader_election.cc:302] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:34.665360 11908 raft_consensus.cc:2781] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.665488 11908 raft_consensus.cc:482] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.665256 11965 consensus_queue.cc:237] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.665589 11908 raft_consensus.cc:3037] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.668255 11908 raft_consensus.cc:504] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.668864 11908 leader_election.cc:288] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:34.668706 9944 catalog_manager.cc:5484] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:34.669229 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdc2a02fa0d42eb93bbbe6213b4723a" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:34.669452 10094 raft_consensus.cc:3037] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.669486 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2fdc2a02fa0d42eb93bbbe6213b4723a" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:34.669683 10490 raft_consensus.cc:3037] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.671573 10490 raft_consensus.cc:2454] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:34.671681 10094 raft_consensus.cc:2454] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:34.671967 10292 leader_election.cc:302] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:34.672298 11908 raft_consensus.cc:2781] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.672479 11908 raft_consensus.cc:686] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:34.673228 11908 consensus_queue.cc:237] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:34.676265 9944 catalog_manager.cc:5484] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:34.747979 11948 raft_consensus.cc:482] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:34.748239 11948 raft_consensus.cc:504] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.749042 11948 leader_election.cc:288] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:34.749679 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d6673d883d4e788ad3e7fe020d7713" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:34.749720 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d6673d883d4e788ad3e7fe020d7713" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:34.749946 10226 raft_consensus.cc:2454] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:34.749994 10490 raft_consensus.cc:2454] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:34.750458 10028 leader_election.cc:302] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:34.750747 11948 raft_consensus.cc:2781] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:34.750859 11948 raft_consensus.cc:482] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:34.750948 11948 raft_consensus.cc:3037] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.753154 11948 raft_consensus.cc:504] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.753762 11948 leader_election.cc:288] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:34.754159 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d6673d883d4e788ad3e7fe020d7713" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:34.754189 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "94d6673d883d4e788ad3e7fe020d7713" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:34.754388 10226 raft_consensus.cc:3037] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.754390 10490 raft_consensus.cc:3037] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:34.756388 10490 raft_consensus.cc:2454] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:34.756803 10026 leader_election.cc:302] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:34.757066 11948 raft_consensus.cc:2781] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:34.757495 11948 raft_consensus.cc:686] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:34.757519 10226 raft_consensus.cc:2454] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:34.758181 11948 consensus_queue.cc:237] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:34.766911 9944 catalog_manager.cc:5484] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:35.050802 11963 consensus_queue.cc:1035] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.062827 11965 consensus_queue.cc:1035] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.176695 11919 consensus_queue.cc:1035] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:35.176563 11996 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:35.177136 11996 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:35.187593 11996 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:35.188041 11919 consensus_queue.cc:1035] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_10'
I20240503 02:12:35.318157 11948 consensus_queue.cc:1035] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:35.358592 11994 consensus_queue.cc:1035] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 251.568 ms
time per row: 0.125784 ms
I20240503 02:12:35.612706 9944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:35.623304 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 20366a3e8dcc4996bfdd1d60ef1be077 (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.623440 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 20366a3e8dcc4996bfdd1d60ef1be077 (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.623927 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20366a3e8dcc4996bfdd1d60ef1be077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.623921 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20366a3e8dcc4996bfdd1d60ef1be077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.625046 10469 tablet_service.cc:1467] Processing CreateTablet for tablet fc0a049eeb0047558d3c9eb370467c53 (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.625581 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0a049eeb0047558d3c9eb370467c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.625540 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 73ae9bc7a96149259bd807369c2daacd (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.625929 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ae9bc7a96149259bd807369c2daacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.626420 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 20366a3e8dcc4996bfdd1d60ef1be077 (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.626885 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20366a3e8dcc4996bfdd1d60ef1be077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.629236 10072 tablet_service.cc:1467] Processing CreateTablet for tablet fc0a049eeb0047558d3c9eb370467c53 (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.629659 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0a049eeb0047558d3c9eb370467c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.632966 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 73ae9bc7a96149259bd807369c2daacd (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.633090 10338 tablet_service.cc:1467] Processing CreateTablet for tablet fc0a049eeb0047558d3c9eb370467c53 (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.633386 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ae9bc7a96149259bd807369c2daacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.633512 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0a049eeb0047558d3c9eb370467c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.633404 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 73ae9bc7a96149259bd807369c2daacd (DEFAULT_TABLE table=rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:35.633955 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ae9bc7a96149259bd807369c2daacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:35.638391 12060 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:35.639149 12059 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:35.640622 12060 tablet_bootstrap.cc:654] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.641096 12059 tablet_bootstrap.cc:654] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.661792 12063 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:35.664688 12063 tablet_bootstrap.cc:654] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.666222 12064 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:35.668037 12060 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:35.668265 12060 ts_tablet_manager.cc:1367] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.023s
I20240503 02:12:35.668548 12064 tablet_bootstrap.cc:654] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.669312 12060 raft_consensus.cc:348] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.669600 12060 raft_consensus.cc:374] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.669765 12060 raft_consensus.cc:729] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.670104 12060 consensus_queue.cc:260] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.671198 12060 ts_tablet_manager.cc:1398] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:12:35.671308 12059 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:35.671494 12059 ts_tablet_manager.cc:1367] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.032s user 0.012s sys 0.016s
I20240503 02:12:35.671535 12060 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:35.672284 12064 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:35.672515 12064 ts_tablet_manager.cc:1367] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.003s
I20240503 02:12:35.672566 12059 raft_consensus.cc:348] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.672876 12059 raft_consensus.cc:374] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.673027 12059 raft_consensus.cc:729] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.673301 12060 tablet_bootstrap.cc:654] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.673434 12059 consensus_queue.cc:260] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.673441 12064 raft_consensus.cc:348] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.673683 12064 raft_consensus.cc:374] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.673859 12064 raft_consensus.cc:729] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.674155 12064 consensus_queue.cc:260] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.674301 12059 ts_tablet_manager.cc:1398] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:35.674706 12059 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:35.675784 12064 ts_tablet_manager.cc:1398] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20240503 02:12:35.676149 12064 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:35.678184 12064 tablet_bootstrap.cc:654] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.680547 12063 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:35.680877 12063 ts_tablet_manager.cc:1367] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20240503 02:12:35.684830 12063 raft_consensus.cc:348] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.685250 12063 raft_consensus.cc:374] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.685422 12063 raft_consensus.cc:729] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.685813 12063 consensus_queue.cc:260] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.686444 12063 ts_tablet_manager.cc:1398] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:35.686702 12059 tablet_bootstrap.cc:654] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.687512 12063 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:35.687706 12060 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:35.687912 12060 ts_tablet_manager.cc:1367] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.003s
I20240503 02:12:35.687943 12064 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:35.688216 12064 ts_tablet_manager.cc:1367] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20240503 02:12:35.688827 12060 raft_consensus.cc:348] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.689080 12060 raft_consensus.cc:374] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.689018 12064 raft_consensus.cc:348] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.689159 12060 raft_consensus.cc:729] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.689272 12064 raft_consensus.cc:374] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.689349 12064 raft_consensus.cc:729] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.689445 12060 consensus_queue.cc:260] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.689661 12064 consensus_queue.cc:260] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.690327 12064 ts_tablet_manager.cc:1398] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.690382 12063 tablet_bootstrap.cc:654] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.691068 12060 ts_tablet_manager.cc:1398] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:35.691453 12064 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:35.692960 12064 tablet_bootstrap.cc:654] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:35.693917 12008 raft_consensus.cc:482] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.694061 12008 raft_consensus.cc:504] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.695036 12008 leader_election.cc:288] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:35.696584 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20366a3e8dcc4996bfdd1d60ef1be077" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:35.696895 10092 raft_consensus.cc:2454] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:35.697430 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20366a3e8dcc4996bfdd1d60ef1be077" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:35.697729 10226 raft_consensus.cc:2454] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:35.698199 12063 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:35.698345 12063 ts_tablet_manager.cc:1367] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20240503 02:12:35.698659 10294 leader_election.cc:302] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:35.699015 12008 raft_consensus.cc:2781] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.698983 12063 raft_consensus.cc:348] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.699148 12063 raft_consensus.cc:374] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.699158 12008 raft_consensus.cc:482] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.699234 12063 raft_consensus.cc:729] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.699277 12008 raft_consensus.cc:3037] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.699505 12063 consensus_queue.cc:260] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.699987 12063 ts_tablet_manager.cc:1398] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:35.702178 12059 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:35.702411 12059 ts_tablet_manager.cc:1367] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.001s
I20240503 02:12:35.703342 12059 raft_consensus.cc:348] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.703617 12059 raft_consensus.cc:374] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.703716 12059 raft_consensus.cc:729] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.703991 12059 consensus_queue.cc:260] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.704366 12064 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:35.704530 12064 ts_tablet_manager.cc:1367] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.000s
I20240503 02:12:35.704402 12008 raft_consensus.cc:504] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.704607 12059 ts_tablet_manager.cc:1398] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:35.705040 12008 leader_election.cc:288] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:35.705966 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20366a3e8dcc4996bfdd1d60ef1be077" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:35.706223 10226 raft_consensus.cc:3037] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.707049 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20366a3e8dcc4996bfdd1d60ef1be077" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:35.707257 10092 raft_consensus.cc:3037] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.705431 12064 raft_consensus.cc:348] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.707475 12064 raft_consensus.cc:374] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:35.707582 12064 raft_consensus.cc:729] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:35.707808 12064 consensus_queue.cc:260] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.708282 12064 ts_tablet_manager.cc:1398] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20240503 02:12:35.709285 10226 raft_consensus.cc:2454] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:35.709910 10294 leader_election.cc:302] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:35.710229 12008 raft_consensus.cc:2781] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.710309 10092 raft_consensus.cc:2454] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:35.710391 12008 raft_consensus.cc:686] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:35.710726 12008 consensus_queue.cc:237] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:35.715198 9948 catalog_manager.cc:5484] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:35.843950 12053 raft_consensus.cc:482] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.844175 12053 raft_consensus.cc:504] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.845317 12053 leader_election.cc:288] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:35.845870 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0a049eeb0047558d3c9eb370467c53" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:35.845924 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0a049eeb0047558d3c9eb370467c53" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:35.846123 10357 raft_consensus.cc:2454] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:35.846223 10490 raft_consensus.cc:2454] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:35.846657 10028 leader_election.cc:302] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:35.846968 12053 raft_consensus.cc:2781] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.847105 12053 raft_consensus.cc:482] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.847208 12053 raft_consensus.cc:3037] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.849382 12053 raft_consensus.cc:504] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.850044 12053 leader_election.cc:288] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:35.850575 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0a049eeb0047558d3c9eb370467c53" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:35.850678 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fc0a049eeb0047558d3c9eb370467c53" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:35.850800 10357 raft_consensus.cc:3037] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.850920 10490 raft_consensus.cc:3037] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.853292 10490 raft_consensus.cc:2454] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:35.853438 10357 raft_consensus.cc:2454] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:35.853760 10026 leader_election.cc:302] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:35.854053 12053 raft_consensus.cc:2781] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.854197 12053 raft_consensus.cc:686] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:35.854493 12053 consensus_queue.cc:237] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:35.855140 12049 raft_consensus.cc:482] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:35.855300 12049 raft_consensus.cc:504] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.856101 12049 leader_election.cc:288] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:35.856686 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ae9bc7a96149259bd807369c2daacd" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:35.856735 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ae9bc7a96149259bd807369c2daacd" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:35.856942 10359 raft_consensus.cc:2454] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:35.856968 10226 raft_consensus.cc:2454] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:35.857424 10426 leader_election.cc:302] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:35.858660 9948 catalog_manager.cc:5484] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:35.859930 12049 raft_consensus.cc:2781] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:35.860064 12049 raft_consensus.cc:482] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:35.860164 12049 raft_consensus.cc:3037] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.862707 12049 raft_consensus.cc:504] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.863269 12049 leader_election.cc:288] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:35.863647 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ae9bc7a96149259bd807369c2daacd" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:35.863869 10359 raft_consensus.cc:3037] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.863790 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "73ae9bc7a96149259bd807369c2daacd" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:35.864027 10226 raft_consensus.cc:3037] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:35.865909 10359 raft_consensus.cc:2454] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:35.866223 10226 raft_consensus.cc:2454] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:35.866290 10426 leader_election.cc:302] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:35.866583 12049 raft_consensus.cc:2781] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:35.866744 12049 raft_consensus.cc:686] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:35.867098 12049 consensus_queue.cc:237] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:35.870739 9948 catalog_manager.cc:5484] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:36.183023 12008 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.190706 11919 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.255734 11808 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.266640 11808 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:36.313494 12075 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:36.313899 12075 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:36.324144 12075 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_11'
I20240503 02:12:36.387578 12053 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:36.396791 11994 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 376.208 ms
time per row: 0.188104 ms
I20240503 02:12:36.888072 9948 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:36.898562 10207 tablet_service.cc:1467] Processing CreateTablet for tablet b757782cdde440a58c2eafea93c7ad04 (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.899178 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b757782cdde440a58c2eafea93c7ad04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.899257 10074 tablet_service.cc:1467] Processing CreateTablet for tablet ad56223da5874609a18d1b640c22e2f2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.899752 10072 tablet_service.cc:1467] Processing CreateTablet for tablet b757782cdde440a58c2eafea93c7ad04 (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.900722 10338 tablet_service.cc:1467] Processing CreateTablet for tablet b757782cdde440a58c2eafea93c7ad04 (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.901059 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad56223da5874609a18d1b640c22e2f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.901091 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 408abb70150b4da3b1c5abab5e2a344b (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.901222 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b757782cdde440a58c2eafea93c7ad04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.901548 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 408abb70150b4da3b1c5abab5e2a344b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.904883 10469 tablet_service.cc:1467] Processing CreateTablet for tablet ad56223da5874609a18d1b640c22e2f2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.905335 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad56223da5874609a18d1b640c22e2f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.906168 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 408abb70150b4da3b1c5abab5e2a344b (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.906615 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 408abb70150b4da3b1c5abab5e2a344b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.908987 10206 tablet_service.cc:1467] Processing CreateTablet for tablet ad56223da5874609a18d1b640c22e2f2 (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.909534 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ad56223da5874609a18d1b640c22e2f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.912374 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 408abb70150b4da3b1c5abab5e2a344b (DEFAULT_TABLE table=rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:36.912847 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 408abb70150b4da3b1c5abab5e2a344b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.913390 12147 tablet_bootstrap.cc:492] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:36.916661 12147 tablet_bootstrap.cc:654] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.917999 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b757782cdde440a58c2eafea93c7ad04. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:36.930877 12147 tablet_bootstrap.cc:492] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:36.931620 12150 tablet_bootstrap.cc:492] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:36.931926 12147 ts_tablet_manager.cc:1367] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20240503 02:12:36.933157 12147 raft_consensus.cc:348] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.933748 12147 raft_consensus.cc:374] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.933924 12147 raft_consensus.cc:729] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.934422 12149 tablet_bootstrap.cc:492] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:36.934334 12147 consensus_queue.cc:260] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.935154 12147 ts_tablet_manager.cc:1398] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:36.935180 12146 tablet_bootstrap.cc:492] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:36.935479 12150 tablet_bootstrap.cc:654] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.935585 12147 tablet_bootstrap.cc:492] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:36.937899 12147 tablet_bootstrap.cc:654] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.938100 12149 tablet_bootstrap.cc:654] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.944514 12150 tablet_bootstrap.cc:492] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:36.944876 12150 ts_tablet_manager.cc:1367] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20240503 02:12:36.946085 12150 raft_consensus.cc:348] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:36.946449 12150 raft_consensus.cc:374] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.946600 12150 raft_consensus.cc:729] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.946755 12149 tablet_bootstrap.cc:492] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:36.946959 12149 ts_tablet_manager.cc:1367] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.013s user 0.011s sys 0.000s
I20240503 02:12:36.946957 12150 consensus_queue.cc:260] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:36.947711 12150 ts_tablet_manager.cc:1398] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:36.947965 12149 raft_consensus.cc:348] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:36.948271 12150 tablet_bootstrap.cc:492] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:36.948259 12149 raft_consensus.cc:374] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.948318 12147 tablet_bootstrap.cc:492] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:36.948370 12149 raft_consensus.cc:729] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.948529 12147 ts_tablet_manager.cc:1367] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.003s
I20240503 02:12:36.948760 12149 consensus_queue.cc:260] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:36.949479 12147 raft_consensus.cc:348] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:36.949728 12147 raft_consensus.cc:374] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.949821 12147 raft_consensus.cc:729] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.950186 12150 tablet_bootstrap.cc:654] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.950098 12147 consensus_queue.cc:260] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:36.952561 12147 ts_tablet_manager.cc:1398] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:36.953051 12149 ts_tablet_manager.cc:1398] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:36.953423 12149 tablet_bootstrap.cc:492] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:36.954173 12146 tablet_bootstrap.cc:654] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.956820 12149 tablet_bootstrap.cc:654] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.963358 12146 tablet_bootstrap.cc:492] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:36.963601 12146 ts_tablet_manager.cc:1367] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.007s
I20240503 02:12:36.963624 12150 tablet_bootstrap.cc:492] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:36.963630 12149 tablet_bootstrap.cc:492] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:36.963820 12150 ts_tablet_manager.cc:1367] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20240503 02:12:36.963860 12149 ts_tablet_manager.cc:1367] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:12:36.964764 12149 raft_consensus.cc:348] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.964783 12150 raft_consensus.cc:348] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.964892 12146 raft_consensus.cc:348] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:36.965029 12150 raft_consensus.cc:374] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.965029 12149 raft_consensus.cc:374] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.965103 12149 raft_consensus.cc:729] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.965103 12150 raft_consensus.cc:729] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.965147 12146 raft_consensus.cc:374] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.965286 12146 raft_consensus.cc:729] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.965386 12150 consensus_queue.cc:260] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.965394 12149 consensus_queue.cc:260] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.965538 12146 consensus_queue.cc:260] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:36.966046 12150 ts_tablet_manager.cc:1398] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:36.966109 12146 ts_tablet_manager.cc:1398] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:36.966352 11808 raft_consensus.cc:482] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:36.966567 12149 ts_tablet_manager.cc:1398] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:36.966496 11808 raft_consensus.cc:504] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.966830 12149 tablet_bootstrap.cc:492] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:36.967646 11808 leader_election.cc:288] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:36.968160 12146 tablet_bootstrap.cc:492] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:36.968197 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "408abb70150b4da3b1c5abab5e2a344b" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:36.968492 10092 raft_consensus.cc:2454] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:36.968611 12149 tablet_bootstrap.cc:654] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.969889 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "408abb70150b4da3b1c5abab5e2a344b" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:36.970161 10357 raft_consensus.cc:2454] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:36.970350 12146 tablet_bootstrap.cc:654] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:36.971153 10426 leader_election.cc:302] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:36.972290 11808 raft_consensus.cc:2781] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:36.972421 11808 raft_consensus.cc:482] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:36.972548 11808 raft_consensus.cc:3037] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.976742 11808 raft_consensus.cc:504] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.978070 11808 leader_election.cc:288] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:36.978399 12146 tablet_bootstrap.cc:492] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:36.978579 12146 ts_tablet_manager.cc:1367] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.001s
I20240503 02:12:36.978526 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "408abb70150b4da3b1c5abab5e2a344b" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:36.978643 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "408abb70150b4da3b1c5abab5e2a344b" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:36.978790 10357 raft_consensus.cc:3037] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.978940 10092 raft_consensus.cc:3037] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:36.980623 12149 tablet_bootstrap.cc:492] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:36.980799 12149 ts_tablet_manager.cc:1367] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.006s
I20240503 02:12:36.981684 12149 raft_consensus.cc:348] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:36.981920 12149 raft_consensus.cc:374] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.981904 10357 raft_consensus.cc:2454] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:36.982013 12149 raft_consensus.cc:729] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.979466 12146 raft_consensus.cc:348] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:36.982357 10092 raft_consensus.cc:2454] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:36.982276 12149 consensus_queue.cc:260] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:36.982560 12146 raft_consensus.cc:374] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:36.982702 12146 raft_consensus.cc:729] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:36.982699 10426 leader_election.cc:302] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:36.983033 12149 ts_tablet_manager.cc:1398] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:36.983033 11808 raft_consensus.cc:2781] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:36.983139 12146 consensus_queue.cc:260] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:36.983680 11808 raft_consensus.cc:686] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:36.983865 12146 ts_tablet_manager.cc:1398] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:36.984020 11808 consensus_queue.cc:237] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:36.988497 9945 catalog_manager.cc:5484] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:37.002801 12127 raft_consensus.cc:482] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.003021 12127 raft_consensus.cc:504] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:37.003896 12127 leader_election.cc:288] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:37.004391 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad56223da5874609a18d1b640c22e2f2" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:37.004434 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad56223da5874609a18d1b640c22e2f2" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:37.004714 10490 raft_consensus.cc:2454] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:37.004766 10227 raft_consensus.cc:2454] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:37.005252 10028 leader_election.cc:302] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:37.005576 12127 raft_consensus.cc:2781] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:37.005720 12127 raft_consensus.cc:482] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:37.005829 12127 raft_consensus.cc:3037] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.008355 12127 raft_consensus.cc:504] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:37.009245 12127 leader_election.cc:288] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:37.009495 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad56223da5874609a18d1b640c22e2f2" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:37.009562 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ad56223da5874609a18d1b640c22e2f2" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:37.009702 10227 raft_consensus.cc:3037] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.009805 10490 raft_consensus.cc:3037] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.011826 10227 raft_consensus.cc:2454] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:37.011922 10490 raft_consensus.cc:2454] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:37.012244 10028 leader_election.cc:302] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:37.012550 12127 raft_consensus.cc:2781] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:37.012710 12127 raft_consensus.cc:686] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:37.013016 12127 consensus_queue.cc:237] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:37.016661 9945 catalog_manager.cc:5484] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:37.277339 11963 raft_consensus.cc:482] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:37.277529 11963 raft_consensus.cc:504] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:37.278396 11963 leader_election.cc:288] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:37.278963 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b757782cdde440a58c2eafea93c7ad04" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:37.278995 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b757782cdde440a58c2eafea93c7ad04" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:37.279182 10357 raft_consensus.cc:2454] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:37.279265 10092 raft_consensus.cc:2454] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:37.279568 10161 leader_election.cc:302] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:37.279853 11963 raft_consensus.cc:2781] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:37.279956 11963 raft_consensus.cc:482] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:37.280071 11963 raft_consensus.cc:3037] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.282474 11963 raft_consensus.cc:504] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:37.283293 11963 leader_election.cc:288] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:37.283700 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b757782cdde440a58c2eafea93c7ad04" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:37.283934 10357 raft_consensus.cc:3037] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.284129 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b757782cdde440a58c2eafea93c7ad04" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:37.284313 10092 raft_consensus.cc:3037] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:37.286232 10092 raft_consensus.cc:2454] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:37.286609 10161 leader_election.cc:302] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:37.286912 11963 raft_consensus.cc:2781] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:37.287060 10357 raft_consensus.cc:2454] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:37.287086 11963 raft_consensus.cc:686] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:37.287369 11963 consensus_queue.cc:237] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:37.291224 9945 catalog_manager.cc:5484] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:37.366984 11808 consensus_queue.cc:1035] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.373641 11808 consensus_queue.cc:1035] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.603646 12125 consensus_queue.cc:1035] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.609673 12125 consensus_queue.cc:1035] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.705001 11965 consensus_queue.cc:1035] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:37.716434 11965 consensus_queue.cc:1035] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:37.797554 12167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:37.798096 12167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:37.811116 12167 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_12'
INSERT report
rows total: 2000
time total: 306.009 ms
time per row: 0.153004 ms
I20240503 02:12:38.304267 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_13"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:38.319386 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa7ed62ebf44431a1a6db26ef73052a (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.319511 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa7ed62ebf44431a1a6db26ef73052a (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.320041 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7ed62ebf44431a1a6db26ef73052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.320044 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7ed62ebf44431a1a6db26ef73052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.320892 10206 tablet_service.cc:1467] Processing CreateTablet for tablet be61de73e08c4998a51f02cafa46d421 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.321328 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be61de73e08c4998a51f02cafa46d421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.321945 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 12da2248f416404da29a4557b90daef7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.322419 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12da2248f416404da29a4557b90daef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.324970 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 5aa7ed62ebf44431a1a6db26ef73052a (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.325457 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7ed62ebf44431a1a6db26ef73052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.325685 10471 tablet_service.cc:1467] Processing CreateTablet for tablet be61de73e08c4998a51f02cafa46d421 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.326073 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be61de73e08c4998a51f02cafa46d421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.326493 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 12da2248f416404da29a4557b90daef7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.326838 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12da2248f416404da29a4557b90daef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.328891 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 12da2248f416404da29a4557b90daef7 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.329306 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12da2248f416404da29a4557b90daef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.332971 10337 tablet_service.cc:1467] Processing CreateTablet for tablet be61de73e08c4998a51f02cafa46d421 (DEFAULT_TABLE table=rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:38.333448 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be61de73e08c4998a51f02cafa46d421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:38.345228 12237 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:38.347535 12237 tablet_bootstrap.cc:654] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.350366 12236 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:38.353724 12237 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:38.354028 12237 ts_tablet_manager.cc:1367] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20240503 02:12:38.355424 12237 raft_consensus.cc:348] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.355796 12237 raft_consensus.cc:374] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.355986 12237 raft_consensus.cc:729] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.356379 12236 tablet_bootstrap.cc:654] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.356415 12237 consensus_queue.cc:260] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.357574 12237 ts_tablet_manager.cc:1398] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:38.360800 12235 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:38.363629 12235 tablet_bootstrap.cc:654] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.365453 12214 raft_consensus.cc:482] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.365693 12214 raft_consensus.cc:504] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.366618 12214 leader_election.cc:288] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:38.367576 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:38.368933 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:38.369655 12241 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
W20240503 02:12:38.369848 10028 leader_election.cc:341] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:38.371531 10028 leader_election.cc:341] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:38.371711 10028 leader_election.cc:302] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7; no voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082
I20240503 02:12:38.372354 12241 tablet_bootstrap.cc:654] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.374118 12236 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:38.374287 12236 ts_tablet_manager.cc:1367] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.010s
I20240503 02:12:38.375339 12236 raft_consensus.cc:348] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.375658 12236 raft_consensus.cc:374] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.375710 12235 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:38.375767 12236 raft_consensus.cc:729] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.375871 12235 ts_tablet_manager.cc:1367] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20240503 02:12:38.376085 12236 consensus_queue.cc:260] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.376824 12235 raft_consensus.cc:348] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.377135 12235 raft_consensus.cc:374] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.377286 12235 raft_consensus.cc:729] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.377779 12214 raft_consensus.cc:2726] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:38.377766 12235 consensus_queue.cc:260] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.378592 12235 ts_tablet_manager.cc:1398] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:12:38.379084 12235 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:38.379413 12236 ts_tablet_manager.cc:1398] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:12:38.379859 12236 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:38.381047 12235 tablet_bootstrap.cc:654] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.381949 12236 tablet_bootstrap.cc:654] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.382191 12241 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:38.382403 12241 ts_tablet_manager.cc:1367] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20240503 02:12:38.383589 12241 raft_consensus.cc:348] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.383858 12241 raft_consensus.cc:374] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.383945 12241 raft_consensus.cc:729] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.384704 12241 consensus_queue.cc:260] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.386694 12241 ts_tablet_manager.cc:1398] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:38.387022 12241 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:38.388934 12241 tablet_bootstrap.cc:654] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.400336 12236 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:38.400492 12241 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:38.400600 12236 ts_tablet_manager.cc:1367] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.021s user 0.015s sys 0.003s
I20240503 02:12:38.400683 12241 ts_tablet_manager.cc:1367] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20240503 02:12:38.400707 12235 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:38.400887 12235 ts_tablet_manager.cc:1367] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.001s
I20240503 02:12:38.401448 12236 raft_consensus.cc:348] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.401685 12236 raft_consensus.cc:374] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.401587 12241 raft_consensus.cc:348] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.401779 12236 raft_consensus.cc:729] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.401845 12241 raft_consensus.cc:374] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.401746 12235 raft_consensus.cc:348] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.401927 12241 raft_consensus.cc:729] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.401984 12235 raft_consensus.cc:374] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.402050 12235 raft_consensus.cc:729] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.402113 12236 consensus_queue.cc:260] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.402264 12241 consensus_queue.cc:260] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.402318 12235 consensus_queue.cc:260] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.402952 12236 ts_tablet_manager.cc:1398] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:38.402962 12235 ts_tablet_manager.cc:1398] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:38.403378 12236 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:38.404731 12241 ts_tablet_manager.cc:1398] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:38.405076 12241 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:38.405335 12236 tablet_bootstrap.cc:654] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.407660 12241 tablet_bootstrap.cc:654] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:38.413635 12236 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:38.413800 12236 ts_tablet_manager.cc:1367] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 02:12:38.414489 12241 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:38.414635 12241 ts_tablet_manager.cc:1367] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.004s
I20240503 02:12:38.414611 12236 raft_consensus.cc:348] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.414839 12236 raft_consensus.cc:374] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.414952 12236 raft_consensus.cc:729] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.415199 12236 consensus_queue.cc:260] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.415280 12241 raft_consensus.cc:348] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.415480 12241 raft_consensus.cc:374] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:38.415560 12241 raft_consensus.cc:729] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:38.415735 12241 consensus_queue.cc:260] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.415835 12236 ts_tablet_manager.cc:1398] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:38.416214 12241 ts_tablet_manager.cc:1398] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:38.507011 12175 raft_consensus.cc:482] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.507283 12175 raft_consensus.cc:504] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.508332 12175 leader_election.cc:288] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:38.508854 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be61de73e08c4998a51f02cafa46d421" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:38.508975 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be61de73e08c4998a51f02cafa46d421" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:38.509117 10491 raft_consensus.cc:2454] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:38.509294 10358 raft_consensus.cc:2454] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:38.509891 10159 leader_election.cc:302] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:38.510223 12175 raft_consensus.cc:2781] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:38.510386 12175 raft_consensus.cc:482] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:38.510500 12175 raft_consensus.cc:3037] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.512987 12175 raft_consensus.cc:504] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.513674 12175 leader_election.cc:288] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:38.514097 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be61de73e08c4998a51f02cafa46d421" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:38.514178 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "be61de73e08c4998a51f02cafa46d421" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:38.514344 10491 raft_consensus.cc:3037] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.514429 10358 raft_consensus.cc:3037] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.517349 10358 raft_consensus.cc:2454] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:38.517365 10491 raft_consensus.cc:2454] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:38.517756 10161 leader_election.cc:302] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:38.518071 12175 raft_consensus.cc:2781] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.518227 12175 raft_consensus.cc:686] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:38.518522 12175 consensus_queue.cc:237] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.522675 9944 catalog_manager.cc:5484] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:38.537005 12174 raft_consensus.cc:482] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.537217 12174 raft_consensus.cc:504] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.537894 12174 leader_election.cc:288] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:38.538393 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa7ed62ebf44431a1a6db26ef73052a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:38.538393 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa7ed62ebf44431a1a6db26ef73052a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:38.538699 10491 raft_consensus.cc:2454] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:38.538661 10358 raft_consensus.cc:2454] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:38.539149 10159 leader_election.cc:302] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:38.539443 12175 raft_consensus.cc:2781] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:38.539562 12175 raft_consensus.cc:482] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:38.539656 12175 raft_consensus.cc:3037] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.542165 12175 raft_consensus.cc:504] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.542847 12175 leader_election.cc:288] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:38.543210 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa7ed62ebf44431a1a6db26ef73052a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:38.543258 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5aa7ed62ebf44431a1a6db26ef73052a" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:38.543478 10491 raft_consensus.cc:3037] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.543478 10358 raft_consensus.cc:3037] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.546506 10358 raft_consensus.cc:2454] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:38.546490 10491 raft_consensus.cc:2454] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:38.546911 10161 leader_election.cc:302] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:38.547207 12175 raft_consensus.cc:2781] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.547369 12175 raft_consensus.cc:686] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:38.547647 12175 consensus_queue.cc:237] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:38.550846 9944 catalog_manager.cc:5484] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:38.629305 11919 raft_consensus.cc:482] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:38.629580 11919 raft_consensus.cc:504] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.630424 11919 leader_election.cc:288] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:38.630944 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:38.631034 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:38.631230 10227 raft_consensus.cc:2454] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:38.631306 10092 raft_consensus.cc:2454] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:38.631676 10294 leader_election.cc:302] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:38.631971 11919 raft_consensus.cc:2781] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:38.632084 11919 raft_consensus.cc:482] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:38.632169 11919 raft_consensus.cc:3037] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.634447 11919 raft_consensus.cc:504] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.635110 11919 leader_election.cc:288] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:38.635654 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:38.635628 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:38.635900 10227 raft_consensus.cc:3037] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.635936 10092 raft_consensus.cc:3037] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:38.639037 10092 raft_consensus.cc:2454] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:38.639416 10294 leader_election.cc:302] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:38.639555 10227 raft_consensus.cc:2454] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:38.639696 11919 raft_consensus.cc:2781] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:38.640202 11919 raft_consensus.cc:686] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:38.640558 11919 consensus_queue.cc:237] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:38.644356 9944 catalog_manager.cc:5484] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:38.921324 12175 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:38.926609 12175 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:39.076858 12252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:39.077298 12252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20240503 02:12:39.079567 12174 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:39.088074 12252 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:39.095584 12175 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.106515 12250 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:39.121280 12250 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_13'
INSERT report
rows total: 2000
time total: 293.628 ms
time per row: 0.146814 ms
I20240503 02:12:39.575512 9945 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:39.587535 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 288181c4acf049d9ac832141162e59bc (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.587625 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 288181c4acf049d9ac832141162e59bc (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.588107 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 288181c4acf049d9ac832141162e59bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.588157 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 288181c4acf049d9ac832141162e59bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.593544 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 41b4e413cfc2460d9c9c05649c5a1ac5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.593595 10471 tablet_service.cc:1467] Processing CreateTablet for tablet e9e25c582fdb488f91083c6f5add142f (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.594094 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b4e413cfc2460d9c9c05649c5a1ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.594496 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e25c582fdb488f91083c6f5add142f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.594588 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 288181c4acf049d9ac832141162e59bc (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.594851 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 41b4e413cfc2460d9c9c05649c5a1ac5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.595048 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 288181c4acf049d9ac832141162e59bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.595252 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b4e413cfc2460d9c9c05649c5a1ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.595185 10339 tablet_service.cc:1467] Processing CreateTablet for tablet e9e25c582fdb488f91083c6f5add142f (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.595692 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e25c582fdb488f91083c6f5add142f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.596843 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 41b4e413cfc2460d9c9c05649c5a1ac5 (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.597275 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 41b4e413cfc2460d9c9c05649c5a1ac5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.601365 10074 tablet_service.cc:1467] Processing CreateTablet for tablet e9e25c582fdb488f91083c6f5add142f (DEFAULT_TABLE table=rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:39.601807 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9e25c582fdb488f91083c6f5add142f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:39.609175 12323 tablet_bootstrap.cc:492] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:39.611663 12323 tablet_bootstrap.cc:654] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.618438 12324 tablet_bootstrap.cc:492] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:39.621563 12324 tablet_bootstrap.cc:654] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.624821 12323 tablet_bootstrap.cc:492] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:39.625154 12323 ts_tablet_manager.cc:1367] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.014s
I20240503 02:12:39.626430 12323 raft_consensus.cc:348] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.626946 12323 raft_consensus.cc:374] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.627137 12323 raft_consensus.cc:729] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.627533 12323 consensus_queue.cc:260] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.628399 12323 ts_tablet_manager.cc:1398] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:39.629945 12286 raft_consensus.cc:482] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.630002 12323 tablet_bootstrap.cc:492] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:39.630111 12286 raft_consensus.cc:504] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.631184 12286 leader_election.cc:288] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:39.631773 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288181c4acf049d9ac832141162e59bc" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
W20240503 02:12:39.632488 10028 leader_election.cc:341] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:39.632910 12323 tablet_bootstrap.cc:654] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:39.636852 10026 leader_election.cc:341] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Not found: Tablet not found: 288181c4acf049d9ac832141162e59bc
I20240503 02:12:39.637075 10026 leader_election.cc:302] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7; no voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082
I20240503 02:12:39.637955 12286 raft_consensus.cc:2726] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:39.640781 12324 tablet_bootstrap.cc:492] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:39.640992 12324 ts_tablet_manager.cc:1367] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.023s user 0.001s sys 0.008s
I20240503 02:12:39.642036 12324 raft_consensus.cc:348] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.642361 12324 raft_consensus.cc:374] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.642482 12324 raft_consensus.cc:729] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.642865 12324 consensus_queue.cc:260] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.643445 12328 tablet_bootstrap.cc:492] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:39.643640 12324 ts_tablet_manager.cc:1398] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:39.644048 12324 tablet_bootstrap.cc:492] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:39.646950 12324 tablet_bootstrap.cc:654] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.649827 12323 tablet_bootstrap.cc:492] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:39.650177 12323 ts_tablet_manager.cc:1367] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.007s
I20240503 02:12:39.650807 12328 tablet_bootstrap.cc:654] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.651504 12323 raft_consensus.cc:348] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.651957 12323 raft_consensus.cc:374] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.652122 12323 raft_consensus.cc:729] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.652580 12323 consensus_queue.cc:260] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.653262 12324 tablet_bootstrap.cc:492] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:39.653481 12324 ts_tablet_manager.cc:1367] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.002s
I20240503 02:12:39.654121 12326 tablet_bootstrap.cc:492] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:39.654304 12324 raft_consensus.cc:348] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.656402 12323 ts_tablet_manager.cc:1398] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:39.656623 12324 raft_consensus.cc:374] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.656785 12324 raft_consensus.cc:729] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.657251 12326 tablet_bootstrap.cc:654] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.657128 12324 consensus_queue.cc:260] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.657990 12323 tablet_bootstrap.cc:492] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:39.659969 12323 tablet_bootstrap.cc:654] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.660924 12324 ts_tablet_manager.cc:1398] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:12:39.661386 12324 tablet_bootstrap.cc:492] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:39.663865 12324 tablet_bootstrap.cc:654] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.666112 12323 tablet_bootstrap.cc:492] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:39.666373 12323 ts_tablet_manager.cc:1367] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20240503 02:12:39.666591 12326 tablet_bootstrap.cc:492] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:39.666836 12326 ts_tablet_manager.cc:1367] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20240503 02:12:39.667641 12328 tablet_bootstrap.cc:492] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:39.667490 12323 raft_consensus.cc:348] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.667754 12323 raft_consensus.cc:374] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.667820 12328 ts_tablet_manager.cc:1367] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20240503 02:12:39.667855 12326 raft_consensus.cc:348] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.668000 12323 raft_consensus.cc:729] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.668175 12326 raft_consensus.cc:374] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.668373 12326 raft_consensus.cc:729] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.668359 12323 consensus_queue.cc:260] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.668733 12326 consensus_queue.cc:260] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.668783 12328 raft_consensus.cc:348] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.668989 12328 raft_consensus.cc:374] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.669086 12328 raft_consensus.cc:729] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.669337 12328 consensus_queue.cc:260] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.669909 12328 ts_tablet_manager.cc:1398] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:39.670225 12328 tablet_bootstrap.cc:492] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:39.670591 12323 ts_tablet_manager.cc:1398] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:39.671945 12328 tablet_bootstrap.cc:654] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:39.675184 12326 ts_tablet_manager.cc:1398] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:12:39.675423 12324 tablet_bootstrap.cc:492] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:39.677472 12324 ts_tablet_manager.cc:1367] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.008s
I20240503 02:12:39.678218 12324 raft_consensus.cc:348] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.678408 12324 raft_consensus.cc:374] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.678544 12324 raft_consensus.cc:729] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.678937 12328 tablet_bootstrap.cc:492] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:39.678833 12324 consensus_queue.cc:260] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.679139 12328 ts_tablet_manager.cc:1367] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20240503 02:12:39.679438 12324 ts_tablet_manager.cc:1398] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:12:39.680079 12328 raft_consensus.cc:348] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.680313 12328 raft_consensus.cc:374] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:39.680405 12328 raft_consensus.cc:729] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:39.680687 12328 consensus_queue.cc:260] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.681362 12328 ts_tablet_manager.cc:1398] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:39.777711 12305 raft_consensus.cc:482] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.777910 12305 raft_consensus.cc:504] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.778705 12305 leader_election.cc:288] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:39.779246 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e25c582fdb488f91083c6f5add142f" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:39.779389 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e25c582fdb488f91083c6f5add142f" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:39.779518 10093 raft_consensus.cc:2454] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:39.779711 10357 raft_consensus.cc:2454] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:39.780110 10426 leader_election.cc:302] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:39.780395 12305 raft_consensus.cc:2781] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.780562 12305 raft_consensus.cc:482] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.780660 12305 raft_consensus.cc:3037] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.783870 12305 raft_consensus.cc:504] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.784605 12305 leader_election.cc:288] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:39.785005 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e25c582fdb488f91083c6f5add142f" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:39.785105 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9e25c582fdb488f91083c6f5add142f" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:39.785238 10093 raft_consensus.cc:3037] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.785378 10357 raft_consensus.cc:3037] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.787626 10357 raft_consensus.cc:2454] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:39.788017 10426 leader_election.cc:302] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:39.788097 10093 raft_consensus.cc:2454] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:39.788458 12305 raft_consensus.cc:2781] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.788645 12305 raft_consensus.cc:686] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:39.788996 12305 consensus_queue.cc:237] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:39.793115 9944 catalog_manager.cc:5484] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:39.803711 12301 raft_consensus.cc:482] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.803939 12301 raft_consensus.cc:504] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.804759 12301 leader_election.cc:288] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:39.805645 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288181c4acf049d9ac832141162e59bc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:39.806119 10093 raft_consensus.cc:2454] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:39.806581 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288181c4acf049d9ac832141162e59bc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:39.806834 10490 raft_consensus.cc:2454] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:39.807088 10161 leader_election.cc:302] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:39.807431 12301 raft_consensus.cc:2781] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.807551 12301 raft_consensus.cc:482] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.807641 12301 raft_consensus.cc:3037] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.810071 12301 raft_consensus.cc:504] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.810704 12301 leader_election.cc:288] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:39.811206 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288181c4acf049d9ac832141162e59bc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:39.811246 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "288181c4acf049d9ac832141162e59bc" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:39.811451 10490 raft_consensus.cc:3037] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.811447 10093 raft_consensus.cc:3037] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.814649 10490 raft_consensus.cc:2454] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:39.814648 10093 raft_consensus.cc:2454] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:39.815081 10161 leader_election.cc:302] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:39.815673 12301 raft_consensus.cc:2781] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.815842 12301 raft_consensus.cc:686] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:39.816147 12301 consensus_queue.cc:237] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.819653 9944 catalog_manager.cc:5484] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:39.948642 12216 raft_consensus.cc:482] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:39.948905 12216 raft_consensus.cc:504] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.949820 12216 leader_election.cc:288] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:39.950524 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b4e413cfc2460d9c9c05649c5a1ac5" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:39.950560 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b4e413cfc2460d9c9c05649c5a1ac5" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:39.950871 10226 raft_consensus.cc:2454] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:39.950874 10490 raft_consensus.cc:2454] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:39.951370 10028 leader_election.cc:302] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:39.951697 12216 raft_consensus.cc:2781] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:39.951833 12216 raft_consensus.cc:482] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:39.951911 12216 raft_consensus.cc:3037] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.954130 12216 raft_consensus.cc:504] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.954710 12216 leader_election.cc:288] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:39.955334 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b4e413cfc2460d9c9c05649c5a1ac5" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:39.955361 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "41b4e413cfc2460d9c9c05649c5a1ac5" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:39.955610 10226 raft_consensus.cc:3037] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.955631 10490 raft_consensus.cc:3037] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:39.958750 10490 raft_consensus.cc:2454] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:39.958775 10226 raft_consensus.cc:2454] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:39.959326 10026 leader_election.cc:302] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:39.959753 12216 raft_consensus.cc:2781] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:39.959933 12216 raft_consensus.cc:686] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:39.960315 12216 consensus_queue.cc:237] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:39.964242 9944 catalog_manager.cc:5484] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:40.274917 12198 consensus_queue.cc:1035] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.292780 12341 consensus_queue.cc:1035] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.320638 12175 consensus_queue.cc:1035] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.334623 12348 consensus_queue.cc:1035] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:40.421111 12339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:40.421722 12339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:40.430363 12339 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:40.444257 12286 consensus_queue.cc:1035] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:40.451977 12286 consensus_queue.cc:1035] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_14'
INSERT report
rows total: 2000
time total: 279.068 ms
time per row: 0.139534 ms
I20240503 02:12:40.900249 9945 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:40.911124 10074 tablet_service.cc:1467] Processing CreateTablet for tablet ee635c5c4d2c4befa0283cb8226f28d7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.911347 10206 tablet_service.cc:1467] Processing CreateTablet for tablet ee635c5c4d2c4befa0283cb8226f28d7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.911764 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee635c5c4d2c4befa0283cb8226f28d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.911836 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee635c5c4d2c4befa0283cb8226f28d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.911728 10339 tablet_service.cc:1467] Processing CreateTablet for tablet ee635c5c4d2c4befa0283cb8226f28d7 (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.912256 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee635c5c4d2c4befa0283cb8226f28d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.913697 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.914140 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.917420 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.917869 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.918545 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.918797 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 35aea508f59f4348b05d93a5afdde4ba (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.918982 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.919312 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35aea508f59f4348b05d93a5afdde4ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.926227 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 35aea508f59f4348b05d93a5afdde4ba (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.926913 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35aea508f59f4348b05d93a5afdde4ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.933122 12408 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:40.935855 12408 tablet_bootstrap.cc:654] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.938571 12409 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:40.941334 12407 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:40.943710 12409 tablet_bootstrap.cc:654] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.946507 12407 tablet_bootstrap.cc:654] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.949812 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 35aea508f59f4348b05d93a5afdde4ba (DEFAULT_TABLE table=rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:40.949949 12406 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:40.950389 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35aea508f59f4348b05d93a5afdde4ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:40.952669 12406 tablet_bootstrap.cc:654] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.959184 12407 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:40.959420 12408 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:40.959470 12407 ts_tablet_manager.cc:1367] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20240503 02:12:40.959743 12408 ts_tablet_manager.cc:1367] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.027s user 0.012s sys 0.012s
I20240503 02:12:40.960500 12407 raft_consensus.cc:348] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:40.960809 12407 raft_consensus.cc:374] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.960950 12407 raft_consensus.cc:729] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.961221 12407 consensus_queue.cc:260] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:40.961318 12408 raft_consensus.cc:348] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:40.961673 12408 raft_consensus.cc:374] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.961823 12408 raft_consensus.cc:729] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.961884 12407 ts_tablet_manager.cc:1398] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:40.962278 12407 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:40.962186 12408 consensus_queue.cc:260] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:40.964053 12408 ts_tablet_manager.cc:1398] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:40.964362 12407 tablet_bootstrap.cc:654] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.964491 12408 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:40.971997 12408 tablet_bootstrap.cc:654] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.972977 12407 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:40.973234 12407 ts_tablet_manager.cc:1367] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.006s
I20240503 02:12:40.973438 12409 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:40.973666 12409 ts_tablet_manager.cc:1367] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.000s
I20240503 02:12:40.974612 12407 raft_consensus.cc:348] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:40.974939 12407 raft_consensus.cc:374] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.975060 12407 raft_consensus.cc:729] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.975348 12407 consensus_queue.cc:260] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:40.976002 12407 ts_tablet_manager.cc:1398] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:40.981077 12409 raft_consensus.cc:348] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:40.981447 12409 raft_consensus.cc:374] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.981604 12409 raft_consensus.cc:729] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.981941 12409 consensus_queue.cc:260] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:40.983338 12408 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:40.983656 12408 ts_tablet_manager.cc:1367] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.006s
I20240503 02:12:40.984597 12409 ts_tablet_manager.cc:1398] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:12:40.984920 12409 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:40.985045 12408 raft_consensus.cc:348] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:40.985455 12408 raft_consensus.cc:374] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.988682 12409 tablet_bootstrap.cc:654] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:40.985600 12408 raft_consensus.cc:729] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.990998 12408 consensus_queue.cc:260] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:40.991334 12406 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:40.991566 12406 ts_tablet_manager.cc:1367] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.020s
I20240503 02:12:40.991735 12409 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:40.991904 12409 ts_tablet_manager.cc:1367] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:12:40.992520 12406 raft_consensus.cc:348] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:40.992637 12409 raft_consensus.cc:348] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:40.992854 12408 ts_tablet_manager.cc:1398] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:12:40.992871 12406 raft_consensus.cc:374] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.992978 12406 raft_consensus.cc:729] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.992990 12409 raft_consensus.cc:374] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:40.993069 12409 raft_consensus.cc:729] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:40.993335 12409 consensus_queue.cc:260] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:40.993355 12406 consensus_queue.cc:260] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:40.994007 12409 ts_tablet_manager.cc:1398] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:40.994423 12409 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:40.995199 12406 ts_tablet_manager.cc:1398] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:40.995553 12406 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:40.999642 12406 tablet_bootstrap.cc:654] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.000152 12409 tablet_bootstrap.cc:654] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:41.002131 12409 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:41.002285 12409 ts_tablet_manager.cc:1367] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.002s
I20240503 02:12:41.002929 12409 raft_consensus.cc:348] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:41.003113 12409 raft_consensus.cc:374] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.003217 12409 raft_consensus.cc:729] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.003436 12409 consensus_queue.cc:260] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:41.003988 12409 ts_tablet_manager.cc:1398] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:41.005790 12406 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:41.005995 12406 ts_tablet_manager.cc:1367] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20240503 02:12:41.006804 12406 raft_consensus.cc:348] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:41.007088 12406 raft_consensus.cc:374] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:41.007184 12406 raft_consensus.cc:729] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:41.007447 12406 consensus_queue.cc:260] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:41.008539 12406 ts_tablet_manager.cc:1398] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:41.026878 11919 raft_consensus.cc:482] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.027084 11919 raft_consensus.cc:504] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:41.027868 11919 leader_election.cc:288] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:41.028465 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:41.028411 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:41.028733 10491 raft_consensus.cc:2454] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:41.028770 10093 raft_consensus.cc:2454] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:41.029228 10292 leader_election.cc:302] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:41.029497 11919 raft_consensus.cc:2781] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.029595 11919 raft_consensus.cc:482] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.029673 11919 raft_consensus.cc:3037] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.031700 11919 raft_consensus.cc:504] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:41.032263 11919 leader_election.cc:288] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:41.032888 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:41.032877 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:41.033128 10491 raft_consensus.cc:3037] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.033151 10093 raft_consensus.cc:3037] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.036281 10491 raft_consensus.cc:2454] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:41.036238 10093 raft_consensus.cc:2454] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:41.036864 10292 leader_election.cc:302] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:41.037285 11919 raft_consensus.cc:2781] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.037776 11919 raft_consensus.cc:686] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:41.038136 11919 consensus_queue.cc:237] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:41.042476 9945 catalog_manager.cc:5484] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:41.072814 11919 raft_consensus.cc:482] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.073072 11919 raft_consensus.cc:504] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:41.073761 11919 leader_election.cc:288] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:41.074354 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aea508f59f4348b05d93a5afdde4ba" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:41.074369 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aea508f59f4348b05d93a5afdde4ba" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:41.074653 10491 raft_consensus.cc:2454] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:41.074653 10227 raft_consensus.cc:2454] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:41.075094 10292 leader_election.cc:302] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:41.075424 11919 raft_consensus.cc:2781] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.075513 11919 raft_consensus.cc:482] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.075598 11919 raft_consensus.cc:3037] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.077780 11919 raft_consensus.cc:504] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:41.078374 11919 leader_election.cc:288] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:41.078814 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aea508f59f4348b05d93a5afdde4ba" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:41.078836 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aea508f59f4348b05d93a5afdde4ba" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:41.079075 10491 raft_consensus.cc:3037] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.079077 10227 raft_consensus.cc:3037] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.081588 10491 raft_consensus.cc:2454] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:41.081609 10227 raft_consensus.cc:2454] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:41.082036 10292 leader_election.cc:302] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:41.082319 11919 raft_consensus.cc:2781] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.082541 11919 raft_consensus.cc:686] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:41.082870 11919 consensus_queue.cc:237] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:41.086792 9945 catalog_manager.cc:5484] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:41.115193 11919 raft_consensus.cc:482] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:41.115464 11919 raft_consensus.cc:504] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:41.116240 11919 leader_election.cc:288] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:41.116791 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee635c5c4d2c4befa0283cb8226f28d7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:41.116833 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee635c5c4d2c4befa0283cb8226f28d7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:41.117094 10227 raft_consensus.cc:2454] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:41.117094 10093 raft_consensus.cc:2454] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:41.117522 10294 leader_election.cc:302] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:41.117813 11919 raft_consensus.cc:2781] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:41.117944 11919 raft_consensus.cc:482] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:41.118045 11919 raft_consensus.cc:3037] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.120245 11919 raft_consensus.cc:504] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:41.120920 11919 leader_election.cc:288] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:41.121532 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee635c5c4d2c4befa0283cb8226f28d7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:41.121585 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee635c5c4d2c4befa0283cb8226f28d7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:41.121780 10227 raft_consensus.cc:3037] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.121831 10092 raft_consensus.cc:3037] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:41.123908 10227 raft_consensus.cc:2454] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:41.124054 10092 raft_consensus.cc:2454] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:41.124341 10294 leader_election.cc:302] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:41.124730 11919 raft_consensus.cc:2781] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:41.124914 11919 raft_consensus.cc:686] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:41.125247 11919 consensus_queue.cc:237] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:41.129827 9945 catalog_manager.cc:5484] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:41.514415 11919 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.539767 12428 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.566999 12428 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.574237 11919 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:41.602332 12423 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:41.602805 12423 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:41.611734 12423 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:41.621966 11919 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:41.637035 12428 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_15'
W20240503 02:12:41.946422 10292 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.68:35025, user_credentials={real_user=slave}} blocked reactor thread for 52964.9us
INSERT report
rows total: 2000
time total: 296.123 ms
time per row: 0.148061 ms
I20240503 02:12:42.089805 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:42.101667 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 6af8995538bb41f4bf7d2f7e0b5907d2 (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.101835 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 6af8995538bb41f4bf7d2f7e0b5907d2 (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.102298 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af8995538bb41f4bf7d2f7e0b5907d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.102384 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af8995538bb41f4bf7d2f7e0b5907d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.102399 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 97432e625f864b91bcaabd8bf3400ff8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.102867 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97432e625f864b91bcaabd8bf3400ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.104277 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 97432e625f864b91bcaabd8bf3400ff8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.104768 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97432e625f864b91bcaabd8bf3400ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.106426 10073 tablet_service.cc:1467] Processing CreateTablet for tablet a9fb56b18eb14f588d261ee79ad97eea (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.106819 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fb56b18eb14f588d261ee79ad97eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.107918 10469 tablet_service.cc:1467] Processing CreateTablet for tablet a9fb56b18eb14f588d261ee79ad97eea (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.108333 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fb56b18eb14f588d261ee79ad97eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.109411 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 97432e625f864b91bcaabd8bf3400ff8 (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.109879 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 97432e625f864b91bcaabd8bf3400ff8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.113359 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 6af8995538bb41f4bf7d2f7e0b5907d2 (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.113873 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6af8995538bb41f4bf7d2f7e0b5907d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.116982 10338 tablet_service.cc:1467] Processing CreateTablet for tablet a9fb56b18eb14f588d261ee79ad97eea (DEFAULT_TABLE table=rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:42.117467 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9fb56b18eb14f588d261ee79ad97eea. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:42.125252 12485 tablet_bootstrap.cc:492] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:42.127818 12485 tablet_bootstrap.cc:654] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.142211 12486 tablet_bootstrap.cc:492] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:42.142220 12485 tablet_bootstrap.cc:492] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:42.142441 12485 ts_tablet_manager.cc:1367] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.006s
I20240503 02:12:42.143618 12485 raft_consensus.cc:348] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.144171 12485 raft_consensus.cc:374] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.144234 12488 tablet_bootstrap.cc:492] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:42.144358 12485 raft_consensus.cc:729] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.144785 12485 consensus_queue.cc:260] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.145777 12485 ts_tablet_manager.cc:1398] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.002s
I20240503 02:12:42.146304 12488 tablet_bootstrap.cc:654] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.146309 12485 tablet_bootstrap.cc:492] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:42.147657 12486 tablet_bootstrap.cc:654] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.148905 12485 tablet_bootstrap.cc:654] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.157714 12484 tablet_bootstrap.cc:492] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:42.157878 12485 tablet_bootstrap.cc:492] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:42.158100 12485 ts_tablet_manager.cc:1367] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20240503 02:12:42.159070 12485 raft_consensus.cc:348] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.159361 12485 raft_consensus.cc:374] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.159456 12485 raft_consensus.cc:729] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.159699 12484 tablet_bootstrap.cc:654] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.159742 12485 consensus_queue.cc:260] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.160526 12485 ts_tablet_manager.cc:1398] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:42.164749 12488 tablet_bootstrap.cc:492] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:42.165030 12488 ts_tablet_manager.cc:1367] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.000s
I20240503 02:12:42.165546 12484 tablet_bootstrap.cc:492] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:42.165740 12484 ts_tablet_manager.cc:1367] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20240503 02:12:42.166218 12488 raft_consensus.cc:348] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.166544 12488 raft_consensus.cc:374] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.166692 12488 raft_consensus.cc:729] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.166918 12484 raft_consensus.cc:348] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.167068 12488 consensus_queue.cc:260] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.167235 12484 raft_consensus.cc:374] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.167384 12484 raft_consensus.cc:729] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.167688 12484 consensus_queue.cc:260] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.167901 12488 ts_tablet_manager.cc:1398] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:42.168296 12488 tablet_bootstrap.cc:492] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:42.169265 12486 tablet_bootstrap.cc:492] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:42.169746 12484 ts_tablet_manager.cc:1398] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:42.169749 12486 ts_tablet_manager.cc:1367] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.004s
I20240503 02:12:42.170130 12484 tablet_bootstrap.cc:492] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:42.170513 12488 tablet_bootstrap.cc:654] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.170887 12486 raft_consensus.cc:348] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.171193 12486 raft_consensus.cc:374] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.171324 12486 raft_consensus.cc:729] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.171624 12486 consensus_queue.cc:260] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.172919 12484 tablet_bootstrap.cc:654] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.174844 12486 ts_tablet_manager.cc:1398] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:42.175571 12486 tablet_bootstrap.cc:492] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:42.178623 12486 tablet_bootstrap.cc:654] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.179539 12484 tablet_bootstrap.cc:492] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:42.179757 12484 ts_tablet_manager.cc:1367] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:12:42.180639 12484 raft_consensus.cc:348] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.180944 12484 raft_consensus.cc:374] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.181069 12484 raft_consensus.cc:729] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.181432 12484 consensus_queue.cc:260] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.182096 12484 ts_tablet_manager.cc:1398] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:42.182446 12484 tablet_bootstrap.cc:492] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:42.185148 12484 tablet_bootstrap.cc:654] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:42.185703 12488 tablet_bootstrap.cc:492] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:42.186735 12488 ts_tablet_manager.cc:1367] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.004s
I20240503 02:12:42.187727 12488 raft_consensus.cc:348] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.187997 12488 raft_consensus.cc:374] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.188123 12488 raft_consensus.cc:729] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.188526 12488 consensus_queue.cc:260] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.189306 12488 ts_tablet_manager.cc:1398] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:42.190845 12484 tablet_bootstrap.cc:492] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:42.191044 12484 ts_tablet_manager.cc:1367] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20240503 02:12:42.191897 12484 raft_consensus.cc:348] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.192116 12484 raft_consensus.cc:374] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.192234 12484 raft_consensus.cc:729] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.192481 12484 consensus_queue.cc:260] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.193168 12484 ts_tablet_manager.cc:1398] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:42.193992 12486 tablet_bootstrap.cc:492] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:42.194221 12486 ts_tablet_manager.cc:1367] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.010s
I20240503 02:12:42.195071 12486 raft_consensus.cc:348] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.195279 12486 raft_consensus.cc:374] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:42.195363 12486 raft_consensus.cc:729] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:42.195617 12486 consensus_queue.cc:260] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.196132 12486 ts_tablet_manager.cc:1398] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:42.250624 12433 raft_consensus.cc:482] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.250888 12433 raft_consensus.cc:504] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.251840 12433 leader_election.cc:288] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:42.252400 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97432e625f864b91bcaabd8bf3400ff8" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:42.252434 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97432e625f864b91bcaabd8bf3400ff8" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:42.252717 10491 raft_consensus.cc:2454] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:42.252748 10226 raft_consensus.cc:2454] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:42.253199 10028 leader_election.cc:302] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:42.253502 12433 raft_consensus.cc:2781] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.253624 12433 raft_consensus.cc:482] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.253728 12433 raft_consensus.cc:3037] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.256167 12433 raft_consensus.cc:504] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.256770 12433 leader_election.cc:288] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:42.257277 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97432e625f864b91bcaabd8bf3400ff8" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:42.257308 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "97432e625f864b91bcaabd8bf3400ff8" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:42.257567 10491 raft_consensus.cc:3037] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.257581 10226 raft_consensus.cc:3037] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.260725 10491 raft_consensus.cc:2454] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:42.260707 10226 raft_consensus.cc:2454] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:42.261246 10028 leader_election.cc:302] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:42.261571 12433 raft_consensus.cc:2781] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.261756 12433 raft_consensus.cc:686] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:42.262117 12433 consensus_queue.cc:237] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.267041 9946 catalog_manager.cc:5484] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:42.298390 12460 raft_consensus.cc:482] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.298638 12460 raft_consensus.cc:504] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.299505 12460 leader_election.cc:288] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:42.300643 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb56b18eb14f588d261ee79ad97eea" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:42.300927 10094 raft_consensus.cc:2454] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:42.301379 10426 leader_election.cc:302] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:42.301651 12341 raft_consensus.cc:2781] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.301780 12341 raft_consensus.cc:482] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.301875 12341 raft_consensus.cc:3037] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.302274 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb56b18eb14f588d261ee79ad97eea" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:42.302590 10358 raft_consensus.cc:2454] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:42.304340 12341 raft_consensus.cc:504] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.305023 12341 leader_election.cc:288] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:42.305747 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb56b18eb14f588d261ee79ad97eea" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:42.306088 10358 raft_consensus.cc:3037] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.306521 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9fb56b18eb14f588d261ee79ad97eea" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:42.306780 10094 raft_consensus.cc:3037] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.309479 10358 raft_consensus.cc:2454] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:42.310318 10094 raft_consensus.cc:2454] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:42.310693 10426 leader_election.cc:302] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:42.311065 12341 raft_consensus.cc:2781] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.311244 12341 raft_consensus.cc:686] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:42.311591 12341 consensus_queue.cc:237] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:42.315912 9946 catalog_manager.cc:5484] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:42.348366 12175 raft_consensus.cc:482] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:42.348627 12175 raft_consensus.cc:504] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.349318 12175 leader_election.cc:288] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:42.349848 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af8995538bb41f4bf7d2f7e0b5907d2" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:42.350020 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af8995538bb41f4bf7d2f7e0b5907d2" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:42.350127 10094 raft_consensus.cc:2454] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:42.350304 10358 raft_consensus.cc:2454] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:42.350584 10161 leader_election.cc:302] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:42.350881 12175 raft_consensus.cc:2781] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:42.350973 12175 raft_consensus.cc:482] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:42.351085 12175 raft_consensus.cc:3037] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.353296 12175 raft_consensus.cc:504] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.353868 12175 leader_election.cc:288] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:42.354370 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af8995538bb41f4bf7d2f7e0b5907d2" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:42.354429 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6af8995538bb41f4bf7d2f7e0b5907d2" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:42.354614 10094 raft_consensus.cc:3037] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.354625 10358 raft_consensus.cc:3037] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:42.356904 10094 raft_consensus.cc:2454] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:42.356966 10358 raft_consensus.cc:2454] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:42.357352 10161 leader_election.cc:302] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:42.357784 12175 raft_consensus.cc:2781] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:42.357965 12175 raft_consensus.cc:686] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:42.358311 12175 consensus_queue.cc:237] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:42.361974 9946 catalog_manager.cc:5484] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:42.682230 12432 consensus_queue.cc:1035] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.690296 12433 consensus_queue.cc:1035] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.721729 12341 consensus_queue.cc:1035] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.727967 12460 consensus_queue.cc:1035] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.738902 12175 consensus_queue.cc:1035] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:42.755792 12301 consensus_queue.cc:1035] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:42.810220 12500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:42.810600 12500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:42.819991 12500 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_16'
INSERT report
rows total: 2000
time total: 309.987 ms
time per row: 0.154993 ms
I20240503 02:12:43.308903 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_17"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:43.319933 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 674ea54f2cbd4c438cab5a7723c5a2c6 (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.320114 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 674ea54f2cbd4c438cab5a7723c5a2c6 (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.320567 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ea54f2cbd4c438cab5a7723c5a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.320595 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ea54f2cbd4c438cab5a7723c5a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.325139 10074 tablet_service.cc:1467] Processing CreateTablet for tablet e94a48cad1d648b18d709b46566d810b (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.325485 10207 tablet_service.cc:1467] Processing CreateTablet for tablet e94a48cad1d648b18d709b46566d810b (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.325747 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94a48cad1d648b18d709b46566d810b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.325945 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94a48cad1d648b18d709b46566d810b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.328274 10206 tablet_service.cc:1467] Processing CreateTablet for tablet ceb177d6952e4c978c8be93b5b026952 (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.328792 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb177d6952e4c978c8be93b5b026952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.330454 10339 tablet_service.cc:1467] Processing CreateTablet for tablet e94a48cad1d648b18d709b46566d810b (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.330971 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94a48cad1d648b18d709b46566d810b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.335456 10337 tablet_service.cc:1467] Processing CreateTablet for tablet ceb177d6952e4c978c8be93b5b026952 (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.335938 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb177d6952e4c978c8be93b5b026952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.335893 10469 tablet_service.cc:1467] Processing CreateTablet for tablet ceb177d6952e4c978c8be93b5b026952 (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.336369 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb177d6952e4c978c8be93b5b026952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.340857 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 674ea54f2cbd4c438cab5a7723c5a2c6 (DEFAULT_TABLE table=rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:43.341286 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ea54f2cbd4c438cab5a7723c5a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:43.342449 12570 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:43.344863 12570 tablet_bootstrap.cc:654] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.351482 12571 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:43.353806 12571 tablet_bootstrap.cc:654] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.358740 12573 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:43.361346 12573 tablet_bootstrap.cc:654] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.363516 12571 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:43.363760 12571 ts_tablet_manager.cc:1367] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20240503 02:12:43.365239 12571 raft_consensus.cc:348] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.365562 12569 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:43.365651 12571 raft_consensus.cc:374] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.365832 12571 raft_consensus.cc:729] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.366760 12570 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:43.366977 12570 ts_tablet_manager.cc:1367] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.025s user 0.006s sys 0.010s
I20240503 02:12:43.366977 12571 consensus_queue.cc:260] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.367980 12571 ts_tablet_manager.cc:1398] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:43.367965 12570 raft_consensus.cc:348] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.368249 12570 raft_consensus.cc:374] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.368352 12570 raft_consensus.cc:729] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.368741 12571 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:43.368685 12570 consensus_queue.cc:260] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.369894 12569 tablet_bootstrap.cc:654] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.370615 12571 tablet_bootstrap.cc:654] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.370888 12570 ts_tablet_manager.cc:1398] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:43.378674 12569 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:43.378856 12569 ts_tablet_manager.cc:1367] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.005s
I20240503 02:12:43.379477 12571 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:43.379755 12571 ts_tablet_manager.cc:1367] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20240503 02:12:43.379930 12569 raft_consensus.cc:348] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.380193 12569 raft_consensus.cc:374] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.380300 12569 raft_consensus.cc:729] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.380766 12571 raft_consensus.cc:348] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.380590 12569 consensus_queue.cc:260] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.381063 12571 raft_consensus.cc:374] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.381143 12571 raft_consensus.cc:729] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.381501 12569 ts_tablet_manager.cc:1398] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:43.381788 12569 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:43.383406 12569 tablet_bootstrap.cc:654] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.384126 12571 consensus_queue.cc:260] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.384835 12571 ts_tablet_manager.cc:1398] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:43.385922 12569 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:43.386080 12569 ts_tablet_manager.cc:1367] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:43.386099 12571 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:43.386654 12569 raft_consensus.cc:348] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.386818 12569 raft_consensus.cc:374] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.386898 12569 raft_consensus.cc:729] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.387127 12569 consensus_queue.cc:260] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.387512 12569 ts_tablet_manager.cc:1398] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20240503 02:12:43.387743 12569 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:43.387789 12571 tablet_bootstrap.cc:654] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.389308 12569 tablet_bootstrap.cc:654] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.391376 12571 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:43.391546 12571 ts_tablet_manager.cc:1367] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20240503 02:12:43.391582 12569 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:43.391713 12569 ts_tablet_manager.cc:1367] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:43.392500 12569 raft_consensus.cc:348] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.392679 12573 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:43.392702 12569 raft_consensus.cc:374] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.392782 12569 raft_consensus.cc:729] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.392853 12573 ts_tablet_manager.cc:1367] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.005s
I20240503 02:12:43.393000 12569 consensus_queue.cc:260] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.393522 12569 ts_tablet_manager.cc:1398] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:43.393752 12573 raft_consensus.cc:348] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.393998 12573 raft_consensus.cc:374] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.394109 12573 raft_consensus.cc:729] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.394376 12573 consensus_queue.cc:260] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.394990 12573 ts_tablet_manager.cc:1398] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:43.396621 12573 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:43.396624 12432 raft_consensus.cc:482] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.396819 12432 raft_consensus.cc:504] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.397810 12432 leader_election.cc:288] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:43.393090 12571 raft_consensus.cc:348] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.398201 12571 raft_consensus.cc:374] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.398195 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94a48cad1d648b18d709b46566d810b" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:43.398386 12571 raft_consensus.cc:729] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.398355 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94a48cad1d648b18d709b46566d810b" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:43.398542 10226 raft_consensus.cc:2454] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:43.398730 12573 tablet_bootstrap.cc:654] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:43.398649 12571 consensus_queue.cc:260] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.399094 10028 leader_election.cc:302] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:43.399272 12571 ts_tablet_manager.cc:1398] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20240503 02:12:43.399596 10358 raft_consensus.cc:2454] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:43.400873 12432 raft_consensus.cc:2781] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.400983 12432 raft_consensus.cc:482] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.401116 12432 raft_consensus.cc:3037] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.403115 12573 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:43.403295 12573 ts_tablet_manager.cc:1367] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.004s
I20240503 02:12:43.403685 12432 raft_consensus.cc:504] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.404001 12573 raft_consensus.cc:348] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.404171 12573 raft_consensus.cc:374] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:43.404220 12573 raft_consensus.cc:729] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:43.404362 12432 leader_election.cc:288] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:43.404471 12573 consensus_queue.cc:260] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.404888 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94a48cad1d648b18d709b46566d810b" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:43.405066 10226 raft_consensus.cc:3037] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.405138 12573 ts_tablet_manager.cc:1398] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:12:43.406041 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e94a48cad1d648b18d709b46566d810b" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:43.406275 10358 raft_consensus.cc:3037] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.408104 10226 raft_consensus.cc:2454] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:43.409267 10358 raft_consensus.cc:2454] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:43.410178 10028 leader_election.cc:302] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:43.410507 12432 raft_consensus.cc:2781] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.410750 12432 raft_consensus.cc:686] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:43.411087 12432 consensus_queue.cc:237] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:43.414858 9952 catalog_manager.cc:5484] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:43.444182 12175 raft_consensus.cc:482] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.444473 12175 raft_consensus.cc:504] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.445176 12175 leader_election.cc:288] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:43.445675 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb177d6952e4c978c8be93b5b026952" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:43.445757 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb177d6952e4c978c8be93b5b026952" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:43.445940 10358 raft_consensus.cc:2454] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:43.446056 10490 raft_consensus.cc:2454] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:43.446331 10161 leader_election.cc:302] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:43.446702 12175 raft_consensus.cc:2781] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.446810 12175 raft_consensus.cc:482] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.446919 12175 raft_consensus.cc:3037] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.449075 12175 raft_consensus.cc:504] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.449868 12175 leader_election.cc:288] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:43.450086 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb177d6952e4c978c8be93b5b026952" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:43.450275 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ceb177d6952e4c978c8be93b5b026952" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:43.450383 10358 raft_consensus.cc:3037] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.450496 10490 raft_consensus.cc:3037] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.453224 10358 raft_consensus.cc:2454] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:43.453395 10490 raft_consensus.cc:2454] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:43.453655 10161 leader_election.cc:302] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:43.454108 12175 raft_consensus.cc:2781] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.454258 12175 raft_consensus.cc:686] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:43.454596 12175 consensus_queue.cc:237] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.458412 9952 catalog_manager.cc:5484] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:43.484966 12175 raft_consensus.cc:482] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:43.485146 12175 raft_consensus.cc:504] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.485731 12175 leader_election.cc:288] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:43.486110 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ea54f2cbd4c438cab5a7723c5a2c6" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:43.486130 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ea54f2cbd4c438cab5a7723c5a2c6" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:43.486356 10490 raft_consensus.cc:2454] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:43.486382 10358 raft_consensus.cc:2454] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:43.486755 10159 leader_election.cc:302] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:43.487010 12175 raft_consensus.cc:2781] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:43.487114 12175 raft_consensus.cc:482] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:43.487206 12175 raft_consensus.cc:3037] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.489651 12175 raft_consensus.cc:504] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.490274 12175 leader_election.cc:288] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:43.490672 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ea54f2cbd4c438cab5a7723c5a2c6" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:43.490715 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "674ea54f2cbd4c438cab5a7723c5a2c6" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:43.490947 10358 raft_consensus.cc:3037] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.490936 10490 raft_consensus.cc:3037] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:43.493950 10358 raft_consensus.cc:2454] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:43.493950 10490 raft_consensus.cc:2454] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:43.494331 10159 leader_election.cc:302] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:43.494750 12175 raft_consensus.cc:2781] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:43.494937 12175 raft_consensus.cc:686] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:43.495219 12175 consensus_queue.cc:237] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:43.498507 9952 catalog_manager.cc:5484] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:43.791509 12433 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.817797 12432 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.866778 12175 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:43.873705 12561 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:43.939188 12585 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:43.939635 12585 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:43.948259 12585 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_17'
I20240503 02:12:44.078902 12558 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.095726 12175 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 273.637 ms
time per row: 0.136818 ms
I20240503 02:12:44.419853 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:44.431360 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 85f7e94021d446589390c7fa7f2940d3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.431457 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 85f7e94021d446589390c7fa7f2940d3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.431973 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f7e94021d446589390c7fa7f2940d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.431977 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f7e94021d446589390c7fa7f2940d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.432651 10337 tablet_service.cc:1467] Processing CreateTablet for tablet d946fe77359940c1a5c9d4f2fff6e123 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.433133 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d946fe77359940c1a5c9d4f2fff6e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.436952 10072 tablet_service.cc:1467] Processing CreateTablet for tablet d946fe77359940c1a5c9d4f2fff6e123 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.437433 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d946fe77359940c1a5c9d4f2fff6e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.437827 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 85f7e94021d446589390c7fa7f2940d3 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.438288 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 85f7e94021d446589390c7fa7f2940d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.440917 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 02b547bf800642deb59b168b1497f256 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.441035 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 02b547bf800642deb59b168b1497f256 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.441401 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b547bf800642deb59b168b1497f256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.441501 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b547bf800642deb59b168b1497f256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.445787 12655 tablet_bootstrap.cc:492] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:44.445995 10471 tablet_service.cc:1467] Processing CreateTablet for tablet d946fe77359940c1a5c9d4f2fff6e123 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.446486 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d946fe77359940c1a5c9d4f2fff6e123. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.448683 12655 tablet_bootstrap.cc:654] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.448930 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 02b547bf800642deb59b168b1497f256 (DEFAULT_TABLE table=rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:44.449456 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02b547bf800642deb59b168b1497f256. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:44.457988 12657 tablet_bootstrap.cc:492] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:44.460014 12655 tablet_bootstrap.cc:492] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:44.460227 12657 tablet_bootstrap.cc:654] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.460284 12655 ts_tablet_manager.cc:1367] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.008s
I20240503 02:12:44.461355 12655 raft_consensus.cc:348] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.461724 12655 raft_consensus.cc:374] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.461874 12655 raft_consensus.cc:729] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.462211 12655 consensus_queue.cc:260] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.463035 12655 ts_tablet_manager.cc:1398] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:44.472882 12657 tablet_bootstrap.cc:492] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:44.474372 12657 ts_tablet_manager.cc:1367] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.005s
I20240503 02:12:44.475170 12659 tablet_bootstrap.cc:492] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:44.475648 12657 raft_consensus.cc:348] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.475919 12657 raft_consensus.cc:374] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.476085 12657 raft_consensus.cc:729] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.476404 12657 consensus_queue.cc:260] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.476933 12660 tablet_bootstrap.cc:492] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:44.477317 12657 ts_tablet_manager.cc:1398] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:44.477669 12659 tablet_bootstrap.cc:654] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.477691 12657 tablet_bootstrap.cc:492] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:44.479599 12657 tablet_bootstrap.cc:654] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.483546 12660 tablet_bootstrap.cc:654] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.488265 12657 tablet_bootstrap.cc:492] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:44.488637 12657 ts_tablet_manager.cc:1367] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20240503 02:12:44.489796 12657 raft_consensus.cc:348] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.490180 12657 raft_consensus.cc:374] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.490350 12657 raft_consensus.cc:729] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.490830 12657 consensus_queue.cc:260] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.491647 12657 ts_tablet_manager.cc:1398] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:44.492158 12659 tablet_bootstrap.cc:492] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:44.492327 12657 tablet_bootstrap.cc:492] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:44.492365 12659 ts_tablet_manager.cc:1367] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.017s user 0.015s sys 0.000s
I20240503 02:12:44.493274 12659 raft_consensus.cc:348] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.493479 12659 raft_consensus.cc:374] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.493592 12659 raft_consensus.cc:729] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.493839 12659 consensus_queue.cc:260] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.494212 12657 tablet_bootstrap.cc:654] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.494550 12659 ts_tablet_manager.cc:1398] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:12:44.495278 12659 tablet_bootstrap.cc:492] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:44.496973 12659 tablet_bootstrap.cc:654] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.497925 12660 tablet_bootstrap.cc:492] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:44.499111 12660 ts_tablet_manager.cc:1367] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.004s
I20240503 02:12:44.500164 12660 raft_consensus.cc:348] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.500548 12660 raft_consensus.cc:374] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.500690 12660 raft_consensus.cc:729] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.501026 12660 consensus_queue.cc:260] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.502564 12660 ts_tablet_manager.cc:1398] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:44.502602 12590 raft_consensus.cc:482] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.502954 12660 tablet_bootstrap.cc:492] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:44.502804 12590 raft_consensus.cc:504] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.503616 12590 leader_election.cc:288] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:44.503891 12657 tablet_bootstrap.cc:492] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:44.504120 12657 ts_tablet_manager.cc:1367] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20240503 02:12:44.504748 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d946fe77359940c1a5c9d4f2fff6e123" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:44.505079 10358 raft_consensus.cc:2454] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:44.505805 10028 leader_election.cc:302] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:44.505972 12660 tablet_bootstrap.cc:654] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.506146 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d946fe77359940c1a5c9d4f2fff6e123" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:44.506285 12590 raft_consensus.cc:2781] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.506393 12590 raft_consensus.cc:482] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.506418 10491 raft_consensus.cc:2454] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:44.506485 12590 raft_consensus.cc:3037] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.510041 12659 tablet_bootstrap.cc:492] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:44.510241 12659 ts_tablet_manager.cc:1367] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20240503 02:12:44.510324 12590 raft_consensus.cc:504] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.511116 12659 raft_consensus.cc:348] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.511391 12659 raft_consensus.cc:374] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.511476 12659 raft_consensus.cc:729] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.511729 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d946fe77359940c1a5c9d4f2fff6e123" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:44.511801 12659 consensus_queue.cc:260] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.511848 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d946fe77359940c1a5c9d4f2fff6e123" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:44.511982 10491 raft_consensus.cc:3037] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.512122 10358 raft_consensus.cc:3037] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.512418 12659 ts_tablet_manager.cc:1398] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:44.512845 12659 tablet_bootstrap.cc:492] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:44.514415 12659 tablet_bootstrap.cc:654] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:44.515223 10491 raft_consensus.cc:2454] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:44.515239 10358 raft_consensus.cc:2454] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:44.516016 10028 leader_election.cc:302] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:44.516330 12433 raft_consensus.cc:2781] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.516520 12433 raft_consensus.cc:686] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:44.516717 12659 tablet_bootstrap.cc:492] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:44.516853 12659 ts_tablet_manager.cc:1367] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:12:44.516804 12433 consensus_queue.cc:237] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:44.517774 12659 raft_consensus.cc:348] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.518041 12659 raft_consensus.cc:374] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.518169 12659 raft_consensus.cc:729] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.518417 12659 consensus_queue.cc:260] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.519423 12590 leader_election.cc:288] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:44.519519 12660 tablet_bootstrap.cc:492] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:44.519724 12660 ts_tablet_manager.cc:1367] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20240503 02:12:44.520396 12657 raft_consensus.cc:348] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.520658 12657 raft_consensus.cc:374] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.520774 12657 raft_consensus.cc:729] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.520664 12660 raft_consensus.cc:348] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.520901 12660 raft_consensus.cc:374] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:44.521010 12660 raft_consensus.cc:729] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:44.521008 12657 consensus_queue.cc:260] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.521577 12659 ts_tablet_manager.cc:1398] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:44.521838 12657 ts_tablet_manager.cc:1398] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:44.522408 12660 consensus_queue.cc:260] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.523537 12660 ts_tablet_manager.cc:1398] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:44.535459 9952 catalog_manager.cc:5484] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:44.625526 12341 raft_consensus.cc:482] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.625727 12341 raft_consensus.cc:504] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.626567 12341 leader_election.cc:288] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:44.627252 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b547bf800642deb59b168b1497f256" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:44.627295 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b547bf800642deb59b168b1497f256" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:44.627494 10094 raft_consensus.cc:2454] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:44.627605 10226 raft_consensus.cc:2454] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:44.628085 10426 leader_election.cc:302] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:44.628389 12341 raft_consensus.cc:2781] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.628535 12341 raft_consensus.cc:482] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.628612 12341 raft_consensus.cc:3037] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.630851 12341 raft_consensus.cc:504] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.631407 12341 leader_election.cc:288] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:44.631973 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b547bf800642deb59b168b1497f256" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:44.632030 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02b547bf800642deb59b168b1497f256" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:44.632180 10094 raft_consensus.cc:3037] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.632264 10226 raft_consensus.cc:3037] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.634507 10226 raft_consensus.cc:2454] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:44.634516 10094 raft_consensus.cc:2454] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:44.634936 10426 leader_election.cc:302] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:44.635272 12341 raft_consensus.cc:2781] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.635811 12341 raft_consensus.cc:686] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:44.636329 12341 consensus_queue.cc:237] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:44.640184 9952 catalog_manager.cc:5484] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:44.770653 12433 raft_consensus.cc:482] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:44.770923 12433 raft_consensus.cc:504] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.772042 12433 leader_election.cc:288] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:44.772720 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f7e94021d446589390c7fa7f2940d3" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:44.772825 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f7e94021d446589390c7fa7f2940d3" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:44.773016 10226 raft_consensus.cc:2454] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:44.773114 10490 raft_consensus.cc:2454] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:44.773506 10028 leader_election.cc:302] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:44.773809 12433 raft_consensus.cc:2781] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:44.773916 12433 raft_consensus.cc:482] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:44.773984 12433 raft_consensus.cc:3037] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.775992 12433 raft_consensus.cc:504] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.776858 12433 leader_election.cc:288] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:44.777357 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f7e94021d446589390c7fa7f2940d3" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:44.777453 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "85f7e94021d446589390c7fa7f2940d3" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:44.777626 10226 raft_consensus.cc:3037] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.777685 10490 raft_consensus.cc:3037] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:44.780758 10490 raft_consensus.cc:2454] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:44.780751 10226 raft_consensus.cc:2454] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:44.781306 10028 leader_election.cc:302] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:44.781836 12433 raft_consensus.cc:2781] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:44.782004 12433 raft_consensus.cc:686] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:44.782330 12433 consensus_queue.cc:237] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:44.787791 9952 catalog_manager.cc:5484] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:44.913271 12433 consensus_queue.cc:1035] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:44.924938 12432 consensus_queue.cc:1035] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.126188 12671 consensus_queue.cc:1035] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.136569 12670 consensus_queue.cc:1035] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:45.267596 12674 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:45.268064 12674 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:45.277005 12674 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
I20240503 02:12:45.295790 12433 consensus_queue.cc:1035] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:45.301921 12432 consensus_queue.cc:1035] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
Using table 'rebalance_test_table_18'
INSERT report
rows total: 2000
time total: 237.245 ms
time per row: 0.118622 ms
I20240503 02:12:45.714468 9951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalance_test_table_19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:45.725808 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 49ce473013a942698081c389ace54b32 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.725821 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 49ce473013a942698081c389ace54b32 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.726372 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ce473013a942698081c389ace54b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.726401 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ce473013a942698081c389ace54b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.726423 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 5c762709f66f44b98089e69cbace1570 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.726887 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c762709f66f44b98089e69cbace1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.726401 10339 tablet_service.cc:1467] Processing CreateTablet for tablet 5c762709f66f44b98089e69cbace1570 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.727875 10339 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c762709f66f44b98089e69cbace1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.730433 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 5c762709f66f44b98089e69cbace1570 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.730505 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 731649c884c3461ab562590f012727c5 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.730863 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c762709f66f44b98089e69cbace1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.730937 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 731649c884c3461ab562590f012727c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.730995 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 731649c884c3461ab562590f012727c5 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.731405 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 731649c884c3461ab562590f012727c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.736518 10205 tablet_service.cc:1467] Processing CreateTablet for tablet 49ce473013a942698081c389ace54b32 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.737010 10205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ce473013a942698081c389ace54b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.739018 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 731649c884c3461ab562590f012727c5 (DEFAULT_TABLE table=rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:45.739645 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 731649c884c3461ab562590f012727c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:45.761054 12743 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:45.761397 12742 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:45.763320 12743 tablet_bootstrap.cc:654] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.763646 12742 tablet_bootstrap.cc:654] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.769893 12744 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:45.771369 12741 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:45.772295 12744 tablet_bootstrap.cc:654] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.773401 12742 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:45.773485 12741 tablet_bootstrap.cc:654] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.773597 12742 ts_tablet_manager.cc:1367] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20240503 02:12:45.774267 12744 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:45.774412 12744 ts_tablet_manager.cc:1367] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:45.774605 12742 raft_consensus.cc:348] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:45.774844 12742 raft_consensus.cc:374] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.774928 12742 raft_consensus.cc:729] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.776312 12744 raft_consensus.cc:348] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:45.776619 12744 raft_consensus.cc:374] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.776784 12744 raft_consensus.cc:729] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.777087 12744 consensus_queue.cc:260] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:45.775341 12742 consensus_queue.cc:260] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:45.778002 12742 ts_tablet_manager.cc:1398] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20240503 02:12:45.779271 12742 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:45.782124 12743 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:45.782435 12743 ts_tablet_manager.cc:1367] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20240503 02:12:45.783383 12671 raft_consensus.cc:482] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.783571 12742 tablet_bootstrap.cc:654] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.783452 12743 raft_consensus.cc:348] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.783582 12671 raft_consensus.cc:504] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:45.783876 12743 raft_consensus.cc:374] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.784019 12743 raft_consensus.cc:729] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.784300 12743 consensus_queue.cc:260] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.784965 12743 ts_tablet_manager.cc:1398] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:45.785540 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:45.786504 12743 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:45.788687 12743 tablet_bootstrap.cc:654] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:45.789820 10426 leader_election.cc:341] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.791010 12671 leader_election.cc:288] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:45.791651 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
W20240503 02:12:45.792254 10426 leader_election.cc:341] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:45.793663 12741 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:45.793887 12741 ts_tablet_manager.cc:1367] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20240503 02:12:45.792409 10426 leader_election.cc:302] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ec2f1283994a3089741715f811c850; no voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f
I20240503 02:12:45.794818 12671 raft_consensus.cc:2726] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:45.795387 12741 raft_consensus.cc:348] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:45.795578 12741 raft_consensus.cc:374] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.795681 12741 raft_consensus.cc:729] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.795954 12741 consensus_queue.cc:260] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:45.805529 12743 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:45.806219 12741 ts_tablet_manager.cc:1398] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:12:45.806627 12741 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:45.807260 12744 ts_tablet_manager.cc:1398] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.033s user 0.003s sys 0.000s
I20240503 02:12:45.808925 12743 ts_tablet_manager.cc:1367] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.004s
I20240503 02:12:45.811497 12743 raft_consensus.cc:348] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:45.812019 12743 raft_consensus.cc:374] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.812836 12741 tablet_bootstrap.cc:654] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.812983 12743 raft_consensus.cc:729] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.813334 12743 consensus_queue.cc:260] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:45.814193 12743 ts_tablet_manager.cc:1398] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:12:45.814451 12744 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:45.820271 12744 tablet_bootstrap.cc:654] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.823940 12742 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:45.824221 12742 ts_tablet_manager.cc:1367] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.033s
I20240503 02:12:45.825277 12742 raft_consensus.cc:348] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:45.825546 12742 raft_consensus.cc:374] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.825629 12742 raft_consensus.cc:729] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.825946 12742 consensus_queue.cc:260] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:45.827813 12742 ts_tablet_manager.cc:1398] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:45.832121 12741 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:45.832389 12741 ts_tablet_manager.cc:1367] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20240503 02:12:45.832129 12744 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:45.833701 12744 ts_tablet_manager.cc:1367] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.019s user 0.006s sys 0.000s
I20240503 02:12:45.834225 12741 raft_consensus.cc:348] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:45.834489 12741 raft_consensus.cc:374] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.834589 12741 raft_consensus.cc:729] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.834656 12744 raft_consensus.cc:348] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.834906 12744 raft_consensus.cc:374] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.834839 12741 consensus_queue.cc:260] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:45.835000 12744 raft_consensus.cc:729] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.835279 12744 consensus_queue.cc:260] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.835523 12741 ts_tablet_manager.cc:1398] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:45.835947 12741 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:45.839915 12744 ts_tablet_manager.cc:1398] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:45.845716 12741 tablet_bootstrap.cc:654] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:45.849305 12741 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:45.849520 12741 ts_tablet_manager.cc:1367] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20240503 02:12:45.850399 12741 raft_consensus.cc:348] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.850652 12741 raft_consensus.cc:374] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:45.850739 12741 raft_consensus.cc:729] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:45.850963 12741 consensus_queue.cc:260] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.851557 12741 ts_tablet_manager.cc:1398] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:45.954245 12697 raft_consensus.cc:482] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:45.954443 12697 raft_consensus.cc:504] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.955284 12697 leader_election.cc:288] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:45.955615 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731649c884c3461ab562590f012727c5" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:45.955878 10357 raft_consensus.cc:2454] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:45.955781 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731649c884c3461ab562590f012727c5" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:45.956068 10490 raft_consensus.cc:2454] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:45.956372 10161 leader_election.cc:302] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:45.956760 12697 raft_consensus.cc:2781] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:45.956916 12697 raft_consensus.cc:482] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:45.957022 12697 raft_consensus.cc:3037] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.959478 12697 raft_consensus.cc:504] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.960062 12697 leader_election.cc:288] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:45.960394 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731649c884c3461ab562590f012727c5" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:45.960511 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "731649c884c3461ab562590f012727c5" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:45.960690 10357 raft_consensus.cc:3037] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.960744 10490 raft_consensus.cc:3037] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:45.963785 10490 raft_consensus.cc:2454] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:45.963798 10357 raft_consensus.cc:2454] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:45.964190 10159 leader_election.cc:302] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:45.964461 12175 raft_consensus.cc:2781] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:45.964650 12175 raft_consensus.cc:686] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:45.964964 12175 consensus_queue.cc:237] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:45.968919 9947 catalog_manager.cc:5484] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:46.029398 12432 raft_consensus.cc:482] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.029686 12432 raft_consensus.cc:504] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:46.030428 12432 leader_election.cc:288] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:46.031059 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ce473013a942698081c389ace54b32" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:46.031231 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ce473013a942698081c389ace54b32" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:46.031445 10227 raft_consensus.cc:2454] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:46.031530 10357 raft_consensus.cc:2454] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:46.031982 10028 leader_election.cc:302] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:46.032303 12432 raft_consensus.cc:2781] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.032402 12432 raft_consensus.cc:482] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.032550 12432 raft_consensus.cc:3037] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.034868 12432 raft_consensus.cc:504] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:46.035580 12432 leader_election.cc:288] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:12:46.036074 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ce473013a942698081c389ace54b32" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:46.036152 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "49ce473013a942698081c389ace54b32" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:46.036280 10357 raft_consensus.cc:3037] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.036417 10227 raft_consensus.cc:3037] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.038969 10357 raft_consensus.cc:2454] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:46.039388 10028 leader_election.cc:302] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:12:46.039444 10227 raft_consensus.cc:2454] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:46.040069 12432 raft_consensus.cc:2781] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.040323 12432 raft_consensus.cc:686] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:46.041020 12432 consensus_queue.cc:237] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:12:46.045255 9947 catalog_manager.cc:5484] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:46.298135 12719 raft_consensus.cc:482] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:46.298381 12719 raft_consensus.cc:504] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:46.299096 12719 leader_election.cc:288] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:46.299567 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:46.299608 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:46.299793 10490 raft_consensus.cc:2454] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:46.299880 10094 raft_consensus.cc:2454] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:46.300242 10292 leader_election.cc:302] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:46.300546 12719 raft_consensus.cc:2781] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:46.300665 12719 raft_consensus.cc:482] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:46.300769 12719 raft_consensus.cc:3037] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.303038 12719 raft_consensus.cc:504] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:46.303678 12719 leader_election.cc:288] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:46.304176 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:46.304176 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:46.304409 10490 raft_consensus.cc:3037] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.304410 10094 raft_consensus.cc:3037] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:46.307374 10094 raft_consensus.cc:2454] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:46.307399 10490 raft_consensus.cc:2454] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:46.307906 10292 leader_election.cc:302] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:46.308332 12478 raft_consensus.cc:2781] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:46.308522 12478 raft_consensus.cc:686] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:46.308851 12478 consensus_queue.cc:237] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:46.313189 9947 catalog_manager.cc:5484] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:46.458695 12758 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.465826 12175 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.591840 12727 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.602991 12727 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:12:46.775063 12757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:46.775591 12757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:46.784852 12757 flags.cc:425] Enabled experimental flag: --enable_multi_tenancy=false
Using table 'rebalance_test_table_19'
I20240503 02:12:46.879925 12478 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:46.889952 12719 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
INSERT report
rows total: 2000
time total: 262.178 ms
time per row: 0.131089 ms
I20240503 02:12:47.219738 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.69:33471
--local_ip_for_outbound_sockets=127.4.5.69
--tserver_master_addrs=127.4.5.126:35467
--webserver_port=38057
--webserver_interface=127.4.5.69
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
W20240503 02:12:47.645052 12823 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:47.645496 12823 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:47.645612 12823 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:47.654310 12823 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:47.654457 12823 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:47.654626 12823 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.69
I20240503 02:12:47.666301 12823 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.69:33471
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
--webserver_interface=127.4.5.69
--webserver_port=38057
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.69
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:47.668030 12823 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:47.670204 12823 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:47.683830 12831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:47.684314 12830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:47.684947 12833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:47.685809 12823 server_base.cc:1034] running on GCE node
I20240503 02:12:47.686487 12823 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:47.687599 12823 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:47.688863 12823 hybrid_clock.cc:648] HybridClock initialized: now 1714702367688796 us; error 47 us; skew 500 ppm
I20240503 02:12:47.689296 12823 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:47.692097 12823 webserver.cc:460] Webserver started at http://127.4.5.69:38057/ using document root <none> and password file <none>
I20240503 02:12:47.693051 12823 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:47.693212 12823 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:47.701246 12823 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20240503 02:12:47.704604 12839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:47.706619 12823 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:12:47.706943 12823 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-zv3f"
I20240503 02:12:47.707923 12823 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:47.730233 12823 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:47.731339 12823 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:47.731670 12823 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:47.733028 12823 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:47.735409 12823 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:47.735541 12823 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20240503 02:12:47.735705 12823 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:47.735793 12823 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:47.793965 12823 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.69:33471
I20240503 02:12:47.793967 12952 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.69:33471 every 8 connection(s)
I20240503 02:12:47.796317 12823 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/data/info.pb
I20240503 02:12:47.799830 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 12823
I20240503 02:12:47.800717 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.70:43707
--local_ip_for_outbound_sockets=127.4.5.70
--tserver_master_addrs=127.4.5.126:35467
--webserver_port=34607
--webserver_interface=127.4.5.70
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:47.831123 12953 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:47.831579 12953 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:47.832998 12953 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:47.835390 9947 ts_manager.cc:194] Re-registered known tserver with Master: e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:47.836906 9947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.69:49851
W20240503 02:12:48.243780 12958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:48.244124 12958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:48.244244 12958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:48.252305 12958 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:48.252491 12958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:48.252655 12958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.70
I20240503 02:12:48.262461 12958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.70:43707
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
--webserver_interface=127.4.5.70
--webserver_port=34607
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.70
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:48.264146 12958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:48.266603 12958 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:48.281107 12964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:48.280814 12963 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:48.282014 12966 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:48.286195 12958 server_base.cc:1034] running on GCE node
I20240503 02:12:48.286865 12958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:48.287886 12958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:48.289098 12958 hybrid_clock.cc:648] HybridClock initialized: now 1714702368289041 us; error 39 us; skew 500 ppm
I20240503 02:12:48.289409 12958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:48.292004 12958 webserver.cc:460] Webserver started at http://127.4.5.70:34607/ using document root <none> and password file <none>
I20240503 02:12:48.292954 12958 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:48.293119 12958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:48.298599 12958 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20240503 02:12:48.301739 12972 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:48.303344 12958 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20240503 02:12:48.303517 12958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
uuid: "8d454a3607d042dc9559ed59c3ccd84b"
format_stamp: "Formatted at 2024-05-03 02:12:14 on dist-test-slave-zv3f"
I20240503 02:12:48.304483 12958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:48.326198 12958 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:48.327179 12958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:48.327479 12958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:48.328537 12958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:48.330426 12958 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:48.330509 12958 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:48.330639 12958 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:48.330719 12958 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:48.382705 12958 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.70:43707
I20240503 02:12:48.382759 13085 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.70:43707 every 8 connection(s)
I20240503 02:12:48.384757 12958 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/data/info.pb
I20240503 02:12:48.387820 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 12958
I20240503 02:12:48.388700 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.71:46557
--local_ip_for_outbound_sockets=127.4.5.71
--tserver_master_addrs=127.4.5.126:35467
--webserver_port=45667
--webserver_interface=127.4.5.71
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:48.401633 13086 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:48.401978 13086 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:48.402966 13086 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:48.405000 9947 ts_manager.cc:194] Re-registered known tserver with Master: 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:48.405983 9947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.70:38425
W20240503 02:12:48.809417 13090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:48.809924 13090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:48.810092 13090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:48.820497 13090 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:48.820681 13090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:48.820855 13090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.71
I20240503 02:12:48.832218 13090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.71:46557
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
--webserver_interface=127.4.5.71
--webserver_port=45667
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.71
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:48.834051 13090 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:48.836599 13090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20240503 02:12:48.840191 12953 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
W20240503 02:12:48.858294 13096 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:48.858742 13097 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:48.859561 13099 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:48.860121 13090 server_base.cc:1034] running on GCE node
I20240503 02:12:48.861014 13090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:48.862440 13090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:48.864584 13090 hybrid_clock.cc:648] HybridClock initialized: now 1714702368864429 us; error 127 us; skew 500 ppm
I20240503 02:12:48.865176 13090 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:48.876858 13090 webserver.cc:460] Webserver started at http://127.4.5.71:45667/ using document root <none> and password file <none>
I20240503 02:12:48.877961 13090 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:48.878147 13090 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:48.885406 13090 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.002s sys 0.001s
I20240503 02:12:48.888958 13106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:48.890763 13090 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20240503 02:12:48.890949 13090 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
uuid: "f93b103af083471b8e8c0376e9148aad"
format_stamp: "Formatted at 2024-05-03 02:12:15 on dist-test-slave-zv3f"
I20240503 02:12:48.892128 13090 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:48.923406 13090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:48.924427 13090 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:48.924837 13090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:48.925999 13090 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:48.930214 13090 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:48.930330 13090 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:48.930469 13090 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:48.930554 13090 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:49.054458 13090 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.71:46557
I20240503 02:12:49.054476 13219 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.71:46557 every 8 connection(s)
I20240503 02:12:49.056789 13090 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/data/info.pb
I20240503 02:12:49.060590 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 13090
I20240503 02:12:49.061600 4117 external_mini_cluster.cc:1326] Running /tmp/dist-test-task48YlKh/build/asan/bin/kudu
/tmp/dist-test-task48YlKh/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/logs
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.4.5.72:33671
--local_ip_for_outbound_sockets=127.4.5.72
--tserver_master_addrs=127.4.5.126:35467
--webserver_port=35929
--webserver_interface=127.4.5.72
--builtin_ntp_servers=127.4.5.84:33619
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--raft_prepare_replacement_before_eviction=true with env {}
I20240503 02:12:49.106168 13221 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:49.106575 13221 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:49.107622 13221 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:49.109589 9947 ts_manager.cc:194] Re-registered known tserver with Master: f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:49.110626 9947 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.71:55629
I20240503 02:12:49.409183 13086 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
W20240503 02:12:49.560403 13227 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20240503 02:12:49.560793 13227 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20240503 02:12:49.560964 13227 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20240503 02:12:49.569228 13227 flags.cc:425] Enabled experimental flag: --raft_prepare_replacement_before_eviction=true
W20240503 02:12:49.569518 13227 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20240503 02:12:49.569727 13227 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.4.5.72
I20240503 02:12:49.581662 13227 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.4.5.84:33619
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data
--fs_wal_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.4.5.72:33671
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
--webserver_interface=127.4.5.72
--webserver_port=35929
--tserver_master_addrs=127.4.5.126:35467
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.4.5.72
--log_dir=/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c39e72a98e124e5a12080bac42105b996594d806
build type FASTDEBUG
built by None at 03 May 2024 01:43:22 UTC on efd1337a2aa3
build id 99
ASAN enabled
I20240503 02:12:49.583518 13227 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20240503 02:12:49.585892 13227 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20240503 02:12:49.603175 13233 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:49.603672 13234 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20240503 02:12:49.604202 13236 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20240503 02:12:49.605073 13227 server_base.cc:1034] running on GCE node
I20240503 02:12:49.605849 13227 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20240503 02:12:49.607151 13227 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20240503 02:12:49.608386 13227 hybrid_clock.cc:648] HybridClock initialized: now 1714702369608256 us; error 122 us; skew 500 ppm
I20240503 02:12:49.608808 13227 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20240503 02:12:49.611547 13227 webserver.cc:460] Webserver started at http://127.4.5.72:35929/ using document root <none> and password file <none>
I20240503 02:12:49.613284 13227 fs_manager.cc:359] Metadata directory not provided
I20240503 02:12:49.613423 13227 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20240503 02:12:49.619179 13227 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20240503 02:12:49.622452 13242 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:49.623975 13227 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:49.624150 13227 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data,/tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
format_stamp: "Formatted at 2024-05-03 02:12:15 on dist-test-slave-zv3f"
I20240503 02:12:49.625159 13227 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
metadata directory: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal
1 data directories: /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20240503 02:12:49.658375 13227 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20240503 02:12:49.660403 13227 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20240503 02:12:49.660782 13227 kserver.cc:163] Server-wide thread pool size limit: 3276
I20240503 02:12:49.661912 13227 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20240503 02:12:49.663733 13227 ts_tablet_manager.cc:559] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20240503 02:12:49.663834 13227 ts_tablet_manager.cc:505] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:49.663982 13227 ts_tablet_manager.cc:590] Registered 0 tablets
I20240503 02:12:49.664063 13227 ts_tablet_manager.cc:569] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20240503 02:12:49.709235 13227 rpc_server.cc:307] RPC server started. Bound to: 127.4.5.72:33671
I20240503 02:12:49.709266 13354 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.4.5.72:33671 every 8 connection(s)
I20240503 02:12:49.711038 13227 server_base.cc:1166] Dumped server information to /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/data/info.pb
I20240503 02:12:49.713477 4117 external_mini_cluster.cc:1388] Started /tmp/dist-test-task48YlKh/build/asan/bin/kudu as pid 13227
I20240503 02:12:49.727847 9947 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:49.764935 13355 heartbeater.cc:347] Connected to a master server at 127.4.5.126:35467
I20240503 02:12:49.765408 13355 heartbeater.cc:464] Registering TS with master...
I20240503 02:12:49.766788 13355 heartbeater.cc:511] Master 127.4.5.126:35467 requested a full tablet report, sending...
I20240503 02:12:49.773277 9951 ts_manager.cc:194] Re-registered known tserver with Master: 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:49.775128 9951 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.4.5.72:40527
W20240503 02:12:49.878840 10161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 70693.8us
I20240503 02:12:49.903254 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8793a0c97249c2aac68f5a39c7f084 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed]), partition=
I20240503 02:12:49.910984 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8793a0c97249c2aac68f5a39c7f084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.914667 13154 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8793a0c97249c2aac68f5a39c7f084 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed]), partition=
I20240503 02:12:49.916122 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8793a0c97249c2aac68f5a39c7f084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.919678 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 5c8793a0c97249c2aac68f5a39c7f084 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed]), partition=
I20240503 02:12:49.921092 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c8793a0c97249c2aac68f5a39c7f084. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:49.947374 13377 tablet_bootstrap.cc:492] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:49.953699 13377 tablet_bootstrap.cc:654] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:49.955169 13377 log.cc:824] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:50.007160 13379 tablet_bootstrap.cc:492] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:50.013693 13377 tablet_bootstrap.cc:492] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:50.014070 13377 ts_tablet_manager.cc:1367] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.067s user 0.029s sys 0.024s
I20240503 02:12:50.015013 13379 tablet_bootstrap.cc:654] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.016332 13379 log.cc:824] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:50.019496 13379 tablet_bootstrap.cc:492] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:50.019837 13379 ts_tablet_manager.cc:1367] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20240503 02:12:50.027709 13377 raft_consensus.cc:348] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.028250 13377 raft_consensus.cc:374] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.031154 13377 raft_consensus.cc:729] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.032734 13383 tablet_bootstrap.cc:492] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:50.035848 13377 consensus_queue.cc:260] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.039184 13383 tablet_bootstrap.cc:654] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.040091 13379 raft_consensus.cc:348] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.041347 13383 log.cc:824] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:50.042485 13379 raft_consensus.cc:374] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.042613 13379 raft_consensus.cc:729] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.044842 13221 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
I20240503 02:12:50.045177 13377 ts_tablet_manager.cc:1398] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.031s user 0.015s sys 0.006s
I20240503 02:12:50.044260 13379 consensus_queue.cc:260] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.051606 13383 tablet_bootstrap.cc:492] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:50.051954 13383 ts_tablet_manager.cc:1367] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.008s
I20240503 02:12:50.061650 13355 heartbeater.cc:503] Master 127.4.5.126:35467 was elected leader, sending a full tablet report...
I20240503 02:12:50.067135 13383 raft_consensus.cc:348] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.067760 13383 raft_consensus.cc:374] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.067905 13383 raft_consensus.cc:729] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.069463 13383 consensus_queue.cc:260] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.078271 13379 ts_tablet_manager.cc:1398] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.058s user 0.014s sys 0.016s
I20240503 02:12:50.079075 13383 ts_tablet_manager.cc:1398] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.027s user 0.014s sys 0.006s
W20240503 02:12:50.100360 13224 tablet.cc:2369] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20240503 02:12:50.138041 13087 tablet.cc:2369] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:50.143608 13386 raft_consensus.cc:482] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:50.143976 13386 raft_consensus.cc:504] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.146435 13386 leader_election.cc:288] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:50.157061 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8793a0c97249c2aac68f5a39c7f084" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:50.157814 13174 raft_consensus.cc:2454] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:50.158854 13246 leader_election.cc:302] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:50.159344 13386 raft_consensus.cc:2781] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:50.159571 13386 raft_consensus.cc:482] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:50.159711 13386 raft_consensus.cc:3037] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.160246 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8793a0c97249c2aac68f5a39c7f084" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:50.160866 13040 raft_consensus.cc:2454] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:50.163198 13386 raft_consensus.cc:504] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.163947 13386 leader_election.cc:288] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:50.164764 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8793a0c97249c2aac68f5a39c7f084" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:50.164959 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c8793a0c97249c2aac68f5a39c7f084" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:50.165036 13040 raft_consensus.cc:3037] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.165254 13174 raft_consensus.cc:3037] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.167485 13040 raft_consensus.cc:2454] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:50.167940 13246 leader_election.cc:302] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:50.168292 13386 raft_consensus.cc:2781] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:50.168695 13174 raft_consensus.cc:2454] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:50.168964 13386 raft_consensus.cc:686] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:12:50.169608 13386 consensus_queue.cc:237] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:50.176878 9947 catalog_manager.cc:5484] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:50.216889 9947 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:50.225914 9947 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 02:12:50.231727 9947 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_0" } modify_external_catalogs: true
I20240503 02:12:50.233459 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 87bb6c6f9ea646a5a26106312a3742eb (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.234011 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bb6c6f9ea646a5a26106312a3742eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.233649 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 87bb6c6f9ea646a5a26106312a3742eb (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.234622 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bb6c6f9ea646a5a26106312a3742eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.237607 13290 tablet_service.cc:1467] Processing CreateTablet for tablet e6bced8dfb104a5eb0db971cd11aad79 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.238620 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bced8dfb104a5eb0db971cd11aad79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.240900 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 918c524462ad41008d2f54f8e954e9bd (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.241297 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918c524462ad41008d2f54f8e954e9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.245339 13383 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
W20240503 02:12:50.245812 13356 tablet.cc:2369] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:50.247828 13383 tablet_bootstrap.cc:654] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.250614 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 918c524462ad41008d2f54f8e954e9bd (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.251786 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918c524462ad41008d2f54f8e954e9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.254626 12888 tablet_service.cc:1467] Processing CreateTablet for tablet 87bb6c6f9ea646a5a26106312a3742eb (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.254988 12887 tablet_service.cc:1467] Processing CreateTablet for tablet e6bced8dfb104a5eb0db971cd11aad79 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.256503 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bced8dfb104a5eb0db971cd11aad79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.257095 13154 tablet_service.cc:1467] Processing CreateTablet for tablet e6bced8dfb104a5eb0db971cd11aad79 (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.257532 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bced8dfb104a5eb0db971cd11aad79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.266503 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bb6c6f9ea646a5a26106312a3742eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.270534 13377 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:50.271471 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 918c524462ad41008d2f54f8e954e9bd (DEFAULT_TABLE table=rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:50.272135 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918c524462ad41008d2f54f8e954e9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:50.272480 13377 tablet_bootstrap.cc:654] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.279222 9947 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet bb9d2573ecc1451894e2c685a9d71675
I20240503 02:12:50.280078 9947 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet f9b74c584e65451cb19fc004ac27f89a
I20240503 02:12:50.280493 9947 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 16d0fa13022744c0a15d86cc341cb083
I20240503 02:12:50.280712 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9d2573ecc1451894e2c685a9d71675 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:50.282470 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d0fa13022744c0a15d86cc341cb083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:50.283968 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b74c584e65451cb19fc004ac27f89a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:50.284260 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d0fa13022744c0a15d86cc341cb083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:50.284188 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9d2573ecc1451894e2c685a9d71675 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:50.284541 10339 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b74c584e65451cb19fc004ac27f89a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:50.285454 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet bb9d2573ecc1451894e2c685a9d71675 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:50.286067 10205 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b74c584e65451cb19fc004ac27f89a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:50.286315 10206 tablet_service.cc:1514] Processing DeleteTablet for tablet 16d0fa13022744c0a15d86cc341cb083 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:50.310636 13397 tablet_replica.cc:330] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:50.311656 13397 raft_consensus.cc:2229] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.312578 13397 raft_consensus.cc:2258] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.314478 13383 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:50.314675 13383 ts_tablet_manager.cc:1367] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.069s user 0.004s sys 0.007s
I20240503 02:12:50.316017 13383 raft_consensus.cc:348] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.316290 13383 raft_consensus.cc:374] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.317587 13383 raft_consensus.cc:729] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.317886 13383 consensus_queue.cc:260] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.308933 13396 tablet_replica.cc:330] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:50.318706 13383 ts_tablet_manager.cc:1398] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20240503 02:12:50.319474 13396 raft_consensus.cc:2229] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:50.321411 13396 raft_consensus.cc:2258] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.329831 13395 tablet_replica.cc:330] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:50.330740 13395 raft_consensus.cc:2229] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:50.331720 13395 raft_consensus.cc:2258] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.332275 9947 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:50.338356 13398 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:50.339625 13397 ts_tablet_manager.cc:1875] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.340111 13394 tablet_replica.cc:330] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:50.345472 13394 raft_consensus.cc:2229] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.345672 13395 ts_tablet_manager.cc:1875] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.346159 13394 raft_consensus.cc:2258] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.346848 13377 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:50.347142 13377 ts_tablet_manager.cc:1367] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.003s
I20240503 02:12:50.348462 13394 ts_tablet_manager.cc:1875] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.348662 13396 ts_tablet_manager.cc:1875] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.349905 13377 raft_consensus.cc:348] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.356274 13398 tablet_bootstrap.cc:654] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.358676 13398 log.cc:824] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Log is configured to *not* fsync() on all Append() calls
I20240503 02:12:50.362301 13377 raft_consensus.cc:374] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.363518 13394 ts_tablet_manager.cc:1888] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.363868 13377 raft_consensus.cc:729] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.369635 13394 log.cc:1196] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/bb9d2573ecc1451894e2c685a9d71675
I20240503 02:12:50.370676 13394 ts_tablet_manager.cc:1909] T bb9d2573ecc1451894e2c685a9d71675 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:50.371011 13377 consensus_queue.cc:260] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.371501 13384 raft_consensus.cc:482] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:50.371745 13384 raft_consensus.cc:504] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.374624 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet bb9d2573ecc1451894e2c685a9d71675 (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.377251 13384 leader_election.cc:288] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:50.379532 13398 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:50.379999 13398 ts_tablet_manager.cc:1367] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.042s user 0.008s sys 0.006s
I20240503 02:12:50.393785 13377 ts_tablet_manager.cc:1398] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.046s user 0.003s sys 0.002s
I20240503 02:12:50.394264 13377 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:50.394490 13394 tablet_replica.cc:330] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:50.394865 13394 raft_consensus.cc:2229] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.395351 13394 raft_consensus.cc:2258] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.397684 13394 ts_tablet_manager.cc:1875] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.409310 13379 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:50.411715 13379 tablet_bootstrap.cc:654] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.427196 13379 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:50.427446 13379 ts_tablet_manager.cc:1367] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20240503 02:12:50.427284 13394 ts_tablet_manager.cc:1888] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.429140 13394 log.cc:1196] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/16d0fa13022744c0a15d86cc341cb083
I20240503 02:12:50.429822 13394 ts_tablet_manager.cc:1909] T 16d0fa13022744c0a15d86cc341cb083 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:50.429683 13379 raft_consensus.cc:348] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.430066 13379 raft_consensus.cc:374] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.430233 13379 raft_consensus.cc:729] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.430642 13379 consensus_queue.cc:260] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.431342 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 16d0fa13022744c0a15d86cc341cb083 (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.431550 13379 ts_tablet_manager.cc:1398] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.000s sys 0.004s
I20240503 02:12:50.431936 13379 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:50.431900 13398 raft_consensus.cc:348] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.432938 13398 raft_consensus.cc:374] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.433121 13398 raft_consensus.cc:729] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.433594 13403 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:50.434474 13398 consensus_queue.cc:260] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.444751 13040 raft_consensus.cc:1261] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:50.448612 13389 consensus_queue.cc:1035] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:50.452306 13377 tablet_bootstrap.cc:654] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.455302 13398 ts_tablet_manager.cc:1398] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.075s user 0.021s sys 0.001s
I20240503 02:12:50.458832 13398 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:50.461087 13398 tablet_bootstrap.cc:654] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.465130 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bced8dfb104a5eb0db971cd11aad79" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
W20240503 02:12:50.466622 13107 leader_election.cc:341] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:50.469151 13174 raft_consensus.cc:1261] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:50.470166 13389 consensus_queue.cc:1035] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:50.477164 13379 tablet_bootstrap.cc:654] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.478843 13397 ts_tablet_manager.cc:1888] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.480825 9945 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 5c8793a0c97249c2aac68f5a39c7f084 (table rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed]). Expected version 1 got 0
I20240503 02:12:50.482910 13395 ts_tablet_manager.cc:1888] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.482947 13415 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:50.483916 13398 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:50.483678 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bced8dfb104a5eb0db971cd11aad79" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
W20240503 02:12:50.485293 13107 leader_election.cc:341] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:50.485293 13396 ts_tablet_manager.cc:1888] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.485541 13396 log.cc:1196] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/bb9d2573ecc1451894e2c685a9d71675
I20240503 02:12:50.485513 13107 leader_election.cc:302] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:50.485961 13384 raft_consensus.cc:2726] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:50.486166 13396 ts_tablet_manager.cc:1909] T bb9d2573ecc1451894e2c685a9d71675 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:50.486550 13398 ts_tablet_manager.cc:1367] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.028s user 0.017s sys 0.000s
I20240503 02:12:50.486642 13387 raft_consensus.cc:482] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:50.486820 13387 raft_consensus.cc:504] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.487557 13415 tablet.cc:1620] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:50.488469 13397 log.cc:1196] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/bb9d2573ecc1451894e2c685a9d71675
I20240503 02:12:50.487828 13398 raft_consensus.cc:348] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.491739 13398 raft_consensus.cc:374] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.491947 13398 raft_consensus.cc:729] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.492429 13398 consensus_queue.cc:260] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.493650 13398 ts_tablet_manager.cc:1398] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:12:50.494058 13398 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:50.488865 13387 leader_election.cc:288] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:50.496730 13398 tablet_bootstrap.cc:654] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.514437 9945 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 5c8793a0c97249c2aac68f5a39c7f084 (table rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed]). Expected version 1 got 0
I20240503 02:12:50.518244 13397 ts_tablet_manager.cc:1909] T bb9d2573ecc1451894e2c685a9d71675 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:50.533725 13379 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:50.534165 13379 ts_tablet_manager.cc:1367] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.102s user 0.005s sys 0.000s
I20240503 02:12:50.536816 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet bb9d2573ecc1451894e2c685a9d71675 (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.538506 9945 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 5c8793a0c97249c2aac68f5a39c7f084 (table rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed]). Expected version 1 got 0
I20240503 02:12:50.538337 13379 raft_consensus.cc:348] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.538722 13379 raft_consensus.cc:374] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.538813 13379 raft_consensus.cc:729] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.541914 13403 tablet_bootstrap.cc:654] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:50.544036 13397 tablet_replica.cc:330] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:50.544426 13397 raft_consensus.cc:2229] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.544863 13397 raft_consensus.cc:2258] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.545481 13395 log.cc:1196] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/f9b74c584e65451cb19fc004ac27f89a
I20240503 02:12:50.546442 13395 ts_tablet_manager.cc:1909] T f9b74c584e65451cb19fc004ac27f89a P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:50.547151 13424 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:50.549203 13396 tablet_replica.cc:330] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:50.549494 13396 raft_consensus.cc:2229] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.549808 13396 raft_consensus.cc:2258] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.550660 13396 ts_tablet_manager.cc:1875] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.550709 9945 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 5c8793a0c97249c2aac68f5a39c7f084 (table rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed]). Expected version 1 got 0
I20240503 02:12:50.551012 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet f9b74c584e65451cb19fc004ac27f89a (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.551338 13377 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:50.551481 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet bb9d2573ecc1451894e2c685a9d71675 (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.551529 13377 ts_tablet_manager.cc:1367] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.157s user 0.002s sys 0.006s
W20240503 02:12:50.552652 12954 tablet.cc:2369] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:50.556387 13397 ts_tablet_manager.cc:1875] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.559595 13379 consensus_queue.cc:260] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.560700 13379 ts_tablet_manager.cc:1398] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20240503 02:12:50.561003 13401 mvcc.cc:205] Tried to move back new op lower bound from 7023420909307981824 to 7023420908219940864. Current Snapshot: MvccSnapshot[applied={T|T < 7023420909307981824}]
I20240503 02:12:50.563045 13398 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:50.563268 13398 ts_tablet_manager.cc:1367] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.069s user 0.021s sys 0.032s
I20240503 02:12:50.563481 13395 tablet_replica.cc:330] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:50.564179 13398 raft_consensus.cc:348] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.564476 13398 raft_consensus.cc:374] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.564576 13398 raft_consensus.cc:729] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.564862 13398 consensus_queue.cc:260] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.565657 13398 ts_tablet_manager.cc:1398] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20240503 02:12:50.566751 13396 ts_tablet_manager.cc:1888] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.567013 13396 log.cc:1196] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9b74c584e65451cb19fc004ac27f89a
I20240503 02:12:50.567533 13396 ts_tablet_manager.cc:1909] T f9b74c584e65451cb19fc004ac27f89a P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:50.568946 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet f9b74c584e65451cb19fc004ac27f89a (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.569204 13426 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:50.569837 9945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed] alter complete (version 1)
I20240503 02:12:50.559087 13377 raft_consensus.cc:348] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.570315 13377 raft_consensus.cc:374] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.570436 13377 raft_consensus.cc:729] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.571239 13403 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:50.571363 13424 tablet.cc:1620] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:12:50.571509 13403 ts_tablet_manager.cc:1367] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.138s user 0.005s sys 0.000s
I20240503 02:12:50.572366 13377 consensus_queue.cc:260] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.577270 13403 raft_consensus.cc:348] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.579272 13426 tablet.cc:1620] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:50.580132 13403 raft_consensus.cc:374] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:50.580562 13403 raft_consensus.cc:729] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:12:50.563793 13395 raft_consensus.cc:2229] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.581359 13377 ts_tablet_manager.cc:1398] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.029s user 0.001s sys 0.003s
I20240503 02:12:50.581241 13403 consensus_queue.cc:260] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.583791 13403 ts_tablet_manager.cc:1398] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20240503 02:12:50.581651 13395 raft_consensus.cc:2258] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.593539 13395 ts_tablet_manager.cc:1875] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.595753 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bb6c6f9ea646a5a26106312a3742eb" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:50.596357 12908 raft_consensus.cc:2454] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:50.600747 13410 mvcc.cc:205] Tried to move back new op lower bound from 7023420909307981824 to 7023420908219940864. Current Snapshot: MvccSnapshot[applied={T|T < 7023420909307981824}]
I20240503 02:12:50.601228 12973 leader_election.cc:302] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:50.601787 13387 raft_consensus.cc:2781] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:50.601943 13387 raft_consensus.cc:482] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:50.602085 13387 raft_consensus.cc:3037] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.610147 13387 raft_consensus.cc:504] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.610769 13397 ts_tablet_manager.cc:1888] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.611258 13387 leader_election.cc:288] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:50.612318 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bb6c6f9ea646a5a26106312a3742eb" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:50.612843 12908 raft_consensus.cc:3037] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.615542 13397 log.cc:1196] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9b74c584e65451cb19fc004ac27f89a
I20240503 02:12:50.616195 12908 raft_consensus.cc:2454] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:50.616279 13397 ts_tablet_manager.cc:1909] T f9b74c584e65451cb19fc004ac27f89a P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:50.617319 12973 leader_election.cc:302] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:50.617795 13387 raft_consensus.cc:2781] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:50.620025 13387 raft_consensus.cc:686] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:50.621073 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bb6c6f9ea646a5a26106312a3742eb" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:12:50.621322 10359 raft_consensus.cc:2454] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:50.621706 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "87bb6c6f9ea646a5a26106312a3742eb" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:50.621796 13397 tablet_replica.cc:330] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:50.621920 10357 raft_consensus.cc:3037] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.622143 13397 raft_consensus.cc:2229] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:50.622982 13397 raft_consensus.cc:2258] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.624138 10357 raft_consensus.cc:2454] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:50.624119 13397 ts_tablet_manager.cc:1875] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.624652 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet f9b74c584e65451cb19fc004ac27f89a (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.625761 13387 consensus_queue.cc:237] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:50.629220 13395 ts_tablet_manager.cc:1888] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.629491 13395 log.cc:1196] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/16d0fa13022744c0a15d86cc341cb083
I20240503 02:12:50.630098 13395 ts_tablet_manager.cc:1909] T 16d0fa13022744c0a15d86cc341cb083 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:50.631940 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 16d0fa13022744c0a15d86cc341cb083 (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.638357 9945 catalog_manager.cc:5484] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:50.639014 13397 ts_tablet_manager.cc:1888] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.639180 13397 log.cc:1196] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/16d0fa13022744c0a15d86cc341cb083
I20240503 02:12:50.639681 13397 ts_tablet_manager.cc:1909] T 16d0fa13022744c0a15d86cc341cb083 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:50.641075 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 16d0fa13022744c0a15d86cc341cb083 (table rebalance_test_table_0 [id=34473780862345e5b1c9d73d9142cc69]) successfully deleted
I20240503 02:12:50.696360 13384 raft_consensus.cc:482] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:50.696617 13384 raft_consensus.cc:504] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.697364 13384 leader_election.cc:288] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:50.697909 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918c524462ad41008d2f54f8e954e9bd" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:50.698038 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918c524462ad41008d2f54f8e954e9bd" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:50.698419 12908 raft_consensus.cc:2454] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:12:50.698601 13310 raft_consensus.cc:2454] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:12:50.699211 13107 leader_election.cc:302] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:50.700264 13384 raft_consensus.cc:2781] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:50.700393 13384 raft_consensus.cc:482] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:50.700548 13384 raft_consensus.cc:3037] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.703547 13384 raft_consensus.cc:504] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.704244 13384 leader_election.cc:288] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:50.704775 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918c524462ad41008d2f54f8e954e9bd" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:50.704869 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "918c524462ad41008d2f54f8e954e9bd" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:50.705034 13310 raft_consensus.cc:3037] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.705129 12908 raft_consensus.cc:3037] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.707553 12908 raft_consensus.cc:2454] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:12:50.707993 13107 leader_election.cc:302] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:50.708034 13310 raft_consensus.cc:2454] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:12:50.708312 13384 raft_consensus.cc:2781] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:50.727387 13384 raft_consensus.cc:686] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:12:50.728516 13384 consensus_queue.cc:237] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:50.733711 9945 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:50.738725 9951 catalog_manager.cc:5484] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 1, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:50.743721 13415 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:50.744333 13415 tablet.cc:1620] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:50.745451 13424 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:50.746007 13424 tablet.cc:1620] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:12:50.752849 13426 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:50.753397 13426 tablet.cc:1620] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:50.767118 9951 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=4c919810bf3e445f9899797f7d7f62ed] alter complete (version 2)
W20240503 02:12:50.769487 10406 tablet.cc:2369] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:50.774128 13389 raft_consensus.cc:482] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:50.774341 13389 raft_consensus.cc:504] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.775357 13389 leader_election.cc:288] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:50.776326 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bced8dfb104a5eb0db971cd11aad79" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:50.776741 13174 raft_consensus.cc:2454] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:50.777370 13246 leader_election.cc:302] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:50.777809 13389 raft_consensus.cc:2781] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:50.777933 13389 raft_consensus.cc:482] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:50.778041 13389 raft_consensus.cc:3037] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.782462 13389 raft_consensus.cc:504] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.783233 13389 leader_election.cc:288] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:50.785908 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bced8dfb104a5eb0db971cd11aad79" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:50.786190 13174 raft_consensus.cc:3037] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.787262 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 02:12:50.787526 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_2" } modify_external_catalogs: true
I20240503 02:12:50.799279 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 48e4714b492d43af93050b4c07ebf0fc
I20240503 02:12:50.805564 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e4714b492d43af93050b4c07ebf0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:50.805999 13395 tablet_replica.cc:330] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:50.806296 13395 raft_consensus.cc:2229] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.806619 13395 raft_consensus.cc:2258] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.807497 13395 ts_tablet_manager.cc:1875] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.807616 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e4714b492d43af93050b4c07ebf0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:50.807974 13394 tablet_replica.cc:330] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:50.808216 13394 raft_consensus.cc:2229] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:50.808967 13394 raft_consensus.cc:2258] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.810032 13394 ts_tablet_manager.cc:1875] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.811163 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 48e4714b492d43af93050b4c07ebf0fc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:50.811612 13397 tablet_replica.cc:330] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:50.811851 13397 raft_consensus.cc:2229] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.812153 13397 raft_consensus.cc:2258] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.812579 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet e4806c1055824b67beb1643e3d42cb22
I20240503 02:12:50.813158 13397 ts_tablet_manager.cc:1875] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.805543 13174 raft_consensus.cc:2454] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:50.815325 13246 leader_election.cc:302] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:50.817150 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet e4806c1055824b67beb1643e3d42cb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:50.817574 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet e4806c1055824b67beb1643e3d42cb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:50.818004 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet befeaccfcfe14e6bb51545a8290cea7a
I20240503 02:12:50.818481 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet e4806c1055824b67beb1643e3d42cb22 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:50.818651 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet befeaccfcfe14e6bb51545a8290cea7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:50.818831 13396 tablet_replica.cc:330] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:50.819088 13396 raft_consensus.cc:2229] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.819350 13396 raft_consensus.cc:2258] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.820238 13396 ts_tablet_manager.cc:1875] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.824692 13389 raft_consensus.cc:2781] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:50.824939 13389 raft_consensus.cc:686] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:12:50.825377 13389 consensus_queue.cc:237] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:50.825656 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet befeaccfcfe14e6bb51545a8290cea7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:50.829573 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet befeaccfcfe14e6bb51545a8290cea7a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:50 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:50.832712 13396 ts_tablet_manager.cc:1888] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.832973 13396 log.cc:1196] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/e4806c1055824b67beb1643e3d42cb22
I20240503 02:12:50.833611 13396 ts_tablet_manager.cc:1909] T e4806c1055824b67beb1643e3d42cb22 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:50.837051 13397 ts_tablet_manager.cc:1888] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.837108 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet e4806c1055824b67beb1643e3d42cb22 (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:50.837371 13397 log.cc:1196] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/48e4714b492d43af93050b4c07ebf0fc
I20240503 02:12:50.837985 13397 ts_tablet_manager.cc:1909] T 48e4714b492d43af93050b4c07ebf0fc P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:50.839437 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 48e4714b492d43af93050b4c07ebf0fc (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:50.839937 13397 tablet_replica.cc:330] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:50.840274 13397 raft_consensus.cc:2229] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:50.840780 13397 raft_consensus.cc:2258] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.841931 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bced8dfb104a5eb0db971cd11aad79" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:50.842257 12908 raft_consensus.cc:2454] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:50.842690 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e6bced8dfb104a5eb0db971cd11aad79" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:50.842993 12907 raft_consensus.cc:3037] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:50.849141 13394 ts_tablet_manager.cc:1888] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.849186 12907 raft_consensus.cc:2454] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:50.850803 13397 ts_tablet_manager.cc:1875] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.854357 9951 catalog_manager.cc:5484] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:50.861846 13394 log.cc:1196] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/48e4714b492d43af93050b4c07ebf0fc
I20240503 02:12:50.862658 13394 ts_tablet_manager.cc:1909] T 48e4714b492d43af93050b4c07ebf0fc P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:50.862876 13395 ts_tablet_manager.cc:1888] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.863066 13395 log.cc:1196] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/48e4714b492d43af93050b4c07ebf0fc
I20240503 02:12:50.863612 13395 ts_tablet_manager.cc:1909] T 48e4714b492d43af93050b4c07ebf0fc P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:50.864712 13395 tablet_replica.cc:330] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:50.864708 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 48e4714b492d43af93050b4c07ebf0fc (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:50.864940 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 48e4714b492d43af93050b4c07ebf0fc (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:50.864981 13395 raft_consensus.cc:2229] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.865247 13395 raft_consensus.cc:2258] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.866035 13395 ts_tablet_manager.cc:1875] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.866386 13394 tablet_replica.cc:330] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:50.866381 13397 ts_tablet_manager.cc:1888] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.866530 13397 log.cc:1196] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/e4806c1055824b67beb1643e3d42cb22
I20240503 02:12:50.866952 13394 raft_consensus.cc:2229] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.867015 13397 ts_tablet_manager.cc:1909] T e4806c1055824b67beb1643e3d42cb22 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:50.869724 13394 raft_consensus.cc:2258] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.869279 9951 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:50.870863 13394 ts_tablet_manager.cc:1875] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.872498 13397 tablet_replica.cc:330] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:50.872649 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet e4806c1055824b67beb1643e3d42cb22 (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:50.872785 13397 raft_consensus.cc:2229] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:50.873157 13397 raft_consensus.cc:2258] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.874903 13397 ts_tablet_manager.cc:1875] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.886391 13415 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:50.887313 13395 ts_tablet_manager.cc:1888] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.887493 13395 log.cc:1196] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/befeaccfcfe14e6bb51545a8290cea7a
I20240503 02:12:50.887966 13395 ts_tablet_manager.cc:1909] T befeaccfcfe14e6bb51545a8290cea7a P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:50.889474 13426 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:50.890499 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet befeaccfcfe14e6bb51545a8290cea7a (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:50.890686 13424 tablet.cc:1722] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:50.902936 13397 ts_tablet_manager.cc:1888] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.903193 13397 log.cc:1196] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/befeaccfcfe14e6bb51545a8290cea7a
I20240503 02:12:50.904562 13397 ts_tablet_manager.cc:1909] T befeaccfcfe14e6bb51545a8290cea7a P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:50.903366 9945 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=4c919810bf3e445f9899797f7d7f62ed] alter complete (version 3)
I20240503 02:12:50.906014 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet befeaccfcfe14e6bb51545a8290cea7a (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:50.937330 13394 ts_tablet_manager.cc:1888] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.937561 13394 log.cc:1196] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/e4806c1055824b67beb1643e3d42cb22
I20240503 02:12:50.938036 13394 ts_tablet_manager.cc:1909] T e4806c1055824b67beb1643e3d42cb22 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:50.939213 13394 tablet_replica.cc:330] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:50.939345 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet e4806c1055824b67beb1643e3d42cb22 (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:50.939489 13394 raft_consensus.cc:2229] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:50.939786 13394 raft_consensus.cc:2258] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:50.940704 13394 ts_tablet_manager.cc:1875] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:50.949685 13394 ts_tablet_manager.cc:1888] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:50.949887 13394 log.cc:1196] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/befeaccfcfe14e6bb51545a8290cea7a
I20240503 02:12:50.950456 13394 ts_tablet_manager.cc:1909] T befeaccfcfe14e6bb51545a8290cea7a P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:50.953032 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet befeaccfcfe14e6bb51545a8290cea7a (table rebalance_test_table_2 [id=963f4219823a4115966c25e5fea3c554]) successfully deleted
I20240503 02:12:51.016224 9946 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:51.016589 9946 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:51.031693 9946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 5c8793a0c97249c2aac68f5a39c7f084
I20240503 02:12:51.035861 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8793a0c97249c2aac68f5a39c7f084 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:12:51.036223 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8793a0c97249c2aac68f5a39c7f084 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:12:51.036828 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c8793a0c97249c2aac68f5a39c7f084 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:12:51.043792 13468 tablet_replica.cc:330] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:12:51.045473 13468 raft_consensus.cc:2229] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.046025 13468 raft_consensus.cc:2258] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.048290 13468 ts_tablet_manager.cc:1875] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.050612 13470 tablet_replica.cc:330] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:12:51.051494 13469 tablet_replica.cc:330] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:12:51.051951 13470 raft_consensus.cc:2229] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:51.052551 13469 raft_consensus.cc:2229] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.053359 13470 raft_consensus.cc:2258] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.063278 13468 ts_tablet_manager.cc:1888] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:51.063694 13468 log.cc:1196] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/5c8793a0c97249c2aac68f5a39c7f084
I20240503 02:12:51.064520 13468 ts_tablet_manager.cc:1909] T 5c8793a0c97249c2aac68f5a39c7f084 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:12:51.066825 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 5c8793a0c97249c2aac68f5a39c7f084 (table rebalancer_dynamic_table_new_name [id=4c919810bf3e445f9899797f7d7f62ed]) successfully deleted
I20240503 02:12:51.066892 13469 raft_consensus.cc:2258] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.070698 13470 ts_tablet_manager.cc:1875] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.076915 13469 ts_tablet_manager.cc:1875] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.090871 13470 ts_tablet_manager.cc:1888] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:51.091128 13470 log.cc:1196] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/5c8793a0c97249c2aac68f5a39c7f084
I20240503 02:12:51.091845 13470 ts_tablet_manager.cc:1909] T 5c8793a0c97249c2aac68f5a39c7f084 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:12:51.099037 13469 ts_tablet_manager.cc:1888] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:51.099277 13469 log.cc:1196] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/5c8793a0c97249c2aac68f5a39c7f084
I20240503 02:12:51.099903 13469 ts_tablet_manager.cc:1909] T 5c8793a0c97249c2aac68f5a39c7f084 P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:12:51.102090 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 5c8793a0c97249c2aac68f5a39c7f084 (table rebalancer_dynamic_table_new_name [id=4c919810bf3e445f9899797f7d7f62ed]) successfully deleted
I20240503 02:12:51.104976 13387 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.105383 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 5c8793a0c97249c2aac68f5a39c7f084 (table rebalancer_dynamic_table_new_name [id=4c919810bf3e445f9899797f7d7f62ed]) successfully deleted
I20240503 02:12:51.116586 13429 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.134151 13384 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.136142 9944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:51.159781 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 23825a55f02c4e0a8bbcd07b147f626d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd]), partition=
I20240503 02:12:51.160290 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23825a55f02c4e0a8bbcd07b147f626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.160277 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 23825a55f02c4e0a8bbcd07b147f626d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd]), partition=
I20240503 02:12:51.160686 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23825a55f02c4e0a8bbcd07b147f626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.165015 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 23825a55f02c4e0a8bbcd07b147f626d (DEFAULT_TABLE table=rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd]), partition=
I20240503 02:12:51.165484 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23825a55f02c4e0a8bbcd07b147f626d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.170567 13484 tablet_bootstrap.cc:492] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:51.170629 13440 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.172036 13484 tablet_bootstrap.cc:654] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.190047 10074 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:12:51.217883 13484 tablet_bootstrap.cc:492] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:51.218125 13484 ts_tablet_manager.cc:1367] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.016s
I20240503 02:12:51.220352 13488 tablet_bootstrap.cc:492] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:51.222256 13484 raft_consensus.cc:348] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.222534 13484 raft_consensus.cc:374] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.222669 13484 raft_consensus.cc:729] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.223086 13484 consensus_queue.cc:260] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.223855 13484 ts_tablet_manager.cc:1398] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:51.242522 10337 tablet_service.cc:1429] Tablet server has 11 leaders and 0 scanners
I20240503 02:12:51.253068 13020 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:51.295168 13488 tablet_bootstrap.cc:654] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.300519 13489 tablet_bootstrap.cc:492] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:51.315495 13489 tablet_bootstrap.cc:654] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.326213 9944 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 02:12:51.326526 9944 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_4" } modify_external_catalogs: true
I20240503 02:12:51.328905 13290 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:51.335114 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 38b1d32c5ca040fcb67aaad68aef60f2
I20240503 02:12:51.335848 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 6441ebd07bac41e6ae31cbb7e95c67af
I20240503 02:12:51.335911 13488 tablet_bootstrap.cc:492] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:51.336257 13488 ts_tablet_manager.cc:1367] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.116s user 0.000s sys 0.017s
I20240503 02:12:51.337181 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b1d32c5ca040fcb67aaad68aef60f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:51.337358 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b1d32c5ca040fcb67aaad68aef60f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:51.337560 13395 tablet_replica.cc:330] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:51.337549 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 38b1d32c5ca040fcb67aaad68aef60f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:51.337710 13397 tablet_replica.cc:330] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:51.337828 13395 raft_consensus.cc:2229] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.337934 13397 raft_consensus.cc:2229] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.338138 13395 raft_consensus.cc:2258] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.338192 13397 raft_consensus.cc:2258] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.338989 13397 ts_tablet_manager.cc:1875] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.338989 13395 ts_tablet_manager.cc:1875] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.339668 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet 6441ebd07bac41e6ae31cbb7e95c67af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:51.341414 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet ee51997a3a0341719b63934754f13564
I20240503 02:12:51.341393 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 6441ebd07bac41e6ae31cbb7e95c67af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:51.341871 13394 tablet_replica.cc:330] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:51.342149 13394 raft_consensus.cc:2229] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.342201 13389 raft_consensus.cc:482] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.342231 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet ee51997a3a0341719b63934754f13564 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:51.342389 13394 raft_consensus.cc:2258] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.342420 13389 raft_consensus.cc:504] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.343227 13394 ts_tablet_manager.cc:1875] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.343282 13389 leader_election.cc:288] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:51.344247 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23825a55f02c4e0a8bbcd07b147f626d" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:51.344738 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet ee51997a3a0341719b63934754f13564 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:36690
W20240503 02:12:51.345165 13243 leader_election.cc:341] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:51.345628 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23825a55f02c4e0a8bbcd07b147f626d" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:51.347503 13395 ts_tablet_manager.cc:1888] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.347705 13395 log.cc:1196] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/38b1d32c5ca040fcb67aaad68aef60f2
I20240503 02:12:51.348711 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 6441ebd07bac41e6ae31cbb7e95c67af with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:51.350181 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet ee51997a3a0341719b63934754f13564 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42260
W20240503 02:12:51.348378 13246 leader_election.cc:341] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:51.350629 13246 leader_election.cc:302] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:51.351179 13416 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.351707 13416 raft_consensus.cc:2726] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:51.356434 13395 ts_tablet_manager.cc:1909] T 38b1d32c5ca040fcb67aaad68aef60f2 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:51.357731 13395 tablet_replica.cc:330] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:51.358011 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 38b1d32c5ca040fcb67aaad68aef60f2 (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.358122 13395 raft_consensus.cc:2229] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.358494 13395 raft_consensus.cc:2258] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.359409 13395 ts_tablet_manager.cc:1875] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.365548 13496 tablet_replica.cc:330] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:51.365922 13496 raft_consensus.cc:2229] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:51.366464 13496 raft_consensus.cc:2258] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.367808 13496 ts_tablet_manager.cc:1875] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.368072 13395 ts_tablet_manager.cc:1888] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.368333 13395 log.cc:1196] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/6441ebd07bac41e6ae31cbb7e95c67af
I20240503 02:12:51.368901 13395 ts_tablet_manager.cc:1909] T 6441ebd07bac41e6ae31cbb7e95c67af P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:51.370450 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 6441ebd07bac41e6ae31cbb7e95c67af (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.371117 13395 tablet_replica.cc:330] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:51.371409 13395 raft_consensus.cc:2229] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.371863 13395 raft_consensus.cc:2258] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.372830 13395 ts_tablet_manager.cc:1875] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.372834 13394 ts_tablet_manager.cc:1888] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.373008 13394 log.cc:1196] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/6441ebd07bac41e6ae31cbb7e95c67af
I20240503 02:12:51.374075 13397 ts_tablet_manager.cc:1888] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.374255 13397 log.cc:1196] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/38b1d32c5ca040fcb67aaad68aef60f2
I20240503 02:12:51.374334 13489 tablet_bootstrap.cc:492] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:51.374543 13489 ts_tablet_manager.cc:1367] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.074s user 0.000s sys 0.007s
I20240503 02:12:51.374773 13397 ts_tablet_manager.cc:1909] T 38b1d32c5ca040fcb67aaad68aef60f2 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:51.376026 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 38b1d32c5ca040fcb67aaad68aef60f2 (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.376538 13488 raft_consensus.cc:348] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.376821 13488 raft_consensus.cc:374] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.375653 13489 raft_consensus.cc:348] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.377432 13489 raft_consensus.cc:374] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.379562 13394 ts_tablet_manager.cc:1909] T 6441ebd07bac41e6ae31cbb7e95c67af P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:51.380896 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 6441ebd07bac41e6ae31cbb7e95c67af (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.376927 13488 raft_consensus.cc:729] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.377583 13489 raft_consensus.cc:729] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.381935 13488 consensus_queue.cc:260] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.382660 13488 ts_tablet_manager.cc:1398] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.046s user 0.002s sys 0.000s
I20240503 02:12:51.384600 13489 consensus_queue.cc:260] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.385298 13489 ts_tablet_manager.cc:1398] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:12:51.389669 13397 tablet_replica.cc:330] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:51.390250 13397 raft_consensus.cc:2229] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:51.390698 13394 tablet_replica.cc:330] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:51.399508 13397 raft_consensus.cc:2258] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.399940 13394 raft_consensus.cc:2229] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:51.401062 13394 raft_consensus.cc:2258] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.401480 13416 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.403263 13397 ts_tablet_manager.cc:1875] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.406859 13394 ts_tablet_manager.cc:1875] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.424254 13395 ts_tablet_manager.cc:1888] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.427320 13395 log.cc:1196] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/ee51997a3a0341719b63934754f13564
I20240503 02:12:51.427980 13395 ts_tablet_manager.cc:1909] T ee51997a3a0341719b63934754f13564 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:51.430418 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet ee51997a3a0341719b63934754f13564 (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.434134 13397 ts_tablet_manager.cc:1888] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.436771 13394 ts_tablet_manager.cc:1888] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.436918 13394 log.cc:1196] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/ee51997a3a0341719b63934754f13564
I20240503 02:12:51.439730 13394 ts_tablet_manager.cc:1909] T ee51997a3a0341719b63934754f13564 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:51.442464 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet ee51997a3a0341719b63934754f13564 (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.444726 13496 ts_tablet_manager.cc:1888] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.444980 13496 log.cc:1196] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/38b1d32c5ca040fcb67aaad68aef60f2
I20240503 02:12:51.464272 13496 ts_tablet_manager.cc:1909] T 38b1d32c5ca040fcb67aaad68aef60f2 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:51.465970 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 38b1d32c5ca040fcb67aaad68aef60f2 (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.466578 13496 tablet_replica.cc:330] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:51.467144 13496 raft_consensus.cc:2229] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.467515 13496 raft_consensus.cc:2258] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.468544 13496 ts_tablet_manager.cc:1875] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.468614 13475 raft_consensus.cc:482] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.468827 13475 raft_consensus.cc:504] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.469169 13397 log.cc:1196] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/6441ebd07bac41e6ae31cbb7e95c67af
I20240503 02:12:51.469754 13397 ts_tablet_manager.cc:1909] T 6441ebd07bac41e6ae31cbb7e95c67af P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:51.469588 9944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:51.471094 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 6441ebd07bac41e6ae31cbb7e95c67af (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.478057 13475 leader_election.cc:288] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:51.484073 13429 raft_consensus.cc:482] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.484297 13429 raft_consensus.cc:504] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.485231 13429 leader_election.cc:288] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:51.488565 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23825a55f02c4e0a8bbcd07b147f626d" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:51.488900 12907 raft_consensus.cc:2454] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:51.491266 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 559ae0e1e196444f9d50420d0f799754 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.491854 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559ae0e1e196444f9d50420d0f799754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.507071 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 559ae0e1e196444f9d50420d0f799754 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.507634 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559ae0e1e196444f9d50420d0f799754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.508328 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 559ae0e1e196444f9d50420d0f799754 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.508899 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559ae0e1e196444f9d50420d0f799754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.508920 13154 tablet_service.cc:1467] Processing CreateTablet for tablet fb944e086c1f4287a1d4b5ec7aa87988 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.509467 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb944e086c1f4287a1d4b5ec7aa87988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.511009 10469 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:12:51.516592 13489 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:51.518653 13489 tablet_bootstrap.cc:654] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.524027 13496 ts_tablet_manager.cc:1888] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.525355 10207 tablet_service.cc:1429] Tablet server has 17 leaders and 0 scanners
I20240503 02:12:51.527655 13496 log.cc:1196] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/ee51997a3a0341719b63934754f13564
I20240503 02:12:51.531158 13496 ts_tablet_manager.cc:1909] T ee51997a3a0341719b63934754f13564 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:51.535444 13020 tablet_service.cc:1467] Processing CreateTablet for tablet fb944e086c1f4287a1d4b5ec7aa87988 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.536038 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb944e086c1f4287a1d4b5ec7aa87988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.537788 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet ee51997a3a0341719b63934754f13564 (table rebalance_test_table_4 [id=acc82f84452148c5843c5adc141fbefb]) successfully deleted
I20240503 02:12:51.539896 13516 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:51.544677 10469 tablet_service.cc:1467] Processing CreateTablet for tablet fb944e086c1f4287a1d4b5ec7aa87988 (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.545207 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb944e086c1f4287a1d4b5ec7aa87988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.548368 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 67be061441154fbba1e227d32474cd5e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.548988 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67be061441154fbba1e227d32474cd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.551570 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 67be061441154fbba1e227d32474cd5e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.552086 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67be061441154fbba1e227d32474cd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.555272 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 67be061441154fbba1e227d32474cd5e (DEFAULT_TABLE table=rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:51.555889 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67be061441154fbba1e227d32474cd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:51.562909 13488 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:51.565910 13488 tablet_bootstrap.cc:654] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.574556 13489 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:51.574802 13489 ts_tablet_manager.cc:1367] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.002s
I20240503 02:12:51.575882 13489 raft_consensus.cc:348] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:51.576182 13489 raft_consensus.cc:374] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.576260 13489 raft_consensus.cc:729] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.577199 12973 leader_election.cc:302] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:51.578397 13387 raft_consensus.cc:2781] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.578523 13387 raft_consensus.cc:482] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.578639 13387 raft_consensus.cc:3037] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.581746 13387 raft_consensus.cc:504] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.583000 13387 leader_election.cc:288] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:51.583688 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23825a55f02c4e0a8bbcd07b147f626d" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:51.583940 12907 raft_consensus.cc:3037] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.585152 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23825a55f02c4e0a8bbcd07b147f626d" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:51.585430 13310 raft_consensus.cc:2454] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:51.586040 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23825a55f02c4e0a8bbcd07b147f626d" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:51.586230 13310 raft_consensus.cc:3037] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.587299 12907 raft_consensus.cc:2454] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:51.589890 13310 raft_consensus.cc:2454] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:51.593273 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23825a55f02c4e0a8bbcd07b147f626d" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:51.593967 13310 raft_consensus.cc:2379] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in current term 1: Already voted for candidate 8d454a3607d042dc9559ed59c3ccd84b in this term.
I20240503 02:12:51.593803 13489 consensus_queue.cc:260] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:51.596133 13488 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:51.596354 13488 ts_tablet_manager.cc:1367] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.002s
I20240503 02:12:51.596700 12973 leader_election.cc:302] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:51.597087 13387 raft_consensus.cc:2781] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.598374 13387 raft_consensus.cc:686] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:51.598713 13387 consensus_queue.cc:237] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:51.602772 13489 ts_tablet_manager.cc:1398] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.028s user 0.002s sys 0.001s
I20240503 02:12:51.604683 13489 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:51.607306 13488 raft_consensus.cc:348] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.607594 13488 raft_consensus.cc:374] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.607668 13488 raft_consensus.cc:729] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.608899 13488 consensus_queue.cc:260] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.609830 13517 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:51.610215 13488 ts_tablet_manager.cc:1398] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:12:51.616027 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23825a55f02c4e0a8bbcd07b147f626d" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:51.617347 13489 tablet_bootstrap.cc:654] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.618513 12843 leader_election.cc:302] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:12:51.621047 13475 raft_consensus.cc:2726] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:51.622077 13153 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:51.622275 13517 tablet_bootstrap.cc:654] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.623257 12886 tablet_service.cc:1429] Tablet server has 0 leaders and 0 scanners
I20240503 02:12:51.640321 13517 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:51.640686 13517 ts_tablet_manager.cc:1367] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.031s user 0.006s sys 0.000s
I20240503 02:12:51.642403 13517 raft_consensus.cc:348] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.644379 13517 raft_consensus.cc:374] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.644629 13517 raft_consensus.cc:729] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.646376 13517 consensus_queue.cc:260] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.650298 13517 ts_tablet_manager.cc:1398] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:12:51.651391 9944 catalog_manager.cc:5484] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:51.659436 13517 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:51.662477 13516 tablet_bootstrap.cc:654] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.669988 13517 tablet_bootstrap.cc:654] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.673457 13489 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:51.673729 13489 ts_tablet_manager.cc:1367] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.070s user 0.001s sys 0.005s
I20240503 02:12:51.684801 13489 raft_consensus.cc:348] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.685143 13489 raft_consensus.cc:374] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.685269 13489 raft_consensus.cc:729] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.685568 13489 consensus_queue.cc:260] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.688350 13489 ts_tablet_manager.cc:1398] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.014s user 0.001s sys 0.002s
I20240503 02:12:51.690294 13516 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:51.690479 13516 ts_tablet_manager.cc:1367] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.151s user 0.005s sys 0.000s
I20240503 02:12:51.693217 13516 raft_consensus.cc:348] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:51.693480 13516 raft_consensus.cc:374] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.693584 13516 raft_consensus.cc:729] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.693904 13516 consensus_queue.cc:260] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:51.694743 13516 ts_tablet_manager.cc:1398] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:51.695050 13516 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:51.706991 13516 tablet_bootstrap.cc:654] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.726213 13516 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:51.726719 13516 ts_tablet_manager.cc:1367] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20240503 02:12:51.728178 13516 raft_consensus.cc:348] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.728569 13516 raft_consensus.cc:374] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.728901 13516 raft_consensus.cc:729] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.729729 13516 consensus_queue.cc:260] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.730521 13516 ts_tablet_manager.cc:1398] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:51.731624 13516 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:51.732244 13517 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:51.732561 13517 ts_tablet_manager.cc:1367] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.073s user 0.008s sys 0.009s
I20240503 02:12:51.733831 13517 raft_consensus.cc:348] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:51.734298 13517 raft_consensus.cc:374] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.734455 13517 raft_consensus.cc:729] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.734980 13517 consensus_queue.cc:260] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:51.735443 13516 tablet_bootstrap.cc:654] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:51.735957 13517 ts_tablet_manager.cc:1398] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:51.736366 13517 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:51.738710 13517 tablet_bootstrap.cc:654] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:51.753049 10538 tablet.cc:2369] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:51.756078 13475 raft_consensus.cc:482] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.756278 13475 raft_consensus.cc:504] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.757426 13475 leader_election.cc:288] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:51.757809 13516 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:51.758018 13516 ts_tablet_manager.cc:1367] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20240503 02:12:51.767196 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be061441154fbba1e227d32474cd5e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:51.767544 10488 raft_consensus.cc:2454] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:51.768313 12840 leader_election.cc:302] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:51.769188 13475 raft_consensus.cc:2781] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.769318 13475 raft_consensus.cc:482] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.769402 13475 raft_consensus.cc:3037] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.769516 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be061441154fbba1e227d32474cd5e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
W20240503 02:12:51.770802 12843 leader_election.cc:341] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:51.772994 13475 raft_consensus.cc:504] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.774833 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be061441154fbba1e227d32474cd5e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:51.774842 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be061441154fbba1e227d32474cd5e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:51.775094 10489 raft_consensus.cc:3037] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:12:51.775611 12843 leader_election.cc:341] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:51.778383 10489 raft_consensus.cc:2454] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:51.779012 12840 leader_election.cc:302] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: f93b103af083471b8e8c0376e9148aad
I20240503 02:12:51.779244 9951 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:51.780409 13475 leader_election.cc:288] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:51.781085 13475 raft_consensus.cc:2781] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.781240 13475 raft_consensus.cc:686] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:12:51.781844 13475 consensus_queue.cc:237] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.788365 13516 raft_consensus.cc:348] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.788708 13516 raft_consensus.cc:374] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.788810 13516 raft_consensus.cc:729] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.789050 13516 consensus_queue.cc:260] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.790103 13517 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:51.790351 13517 ts_tablet_manager.cc:1367] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.015s
I20240503 02:12:51.791289 13517 raft_consensus.cc:348] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.791464 13516 ts_tablet_manager.cc:1398] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.033s user 0.002s sys 0.000s
I20240503 02:12:51.791512 13517 raft_consensus.cc:374] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:51.791604 13517 raft_consensus.cc:729] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:51.791862 13517 consensus_queue.cc:260] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.792567 13517 ts_tablet_manager.cc:1398] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:51.804206 13487 raft_consensus.cc:482] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.804409 13487 raft_consensus.cc:504] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.805451 13487 leader_election.cc:288] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:51.847921 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 02:12:51.848369 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_6" } modify_external_catalogs: true
I20240503 02:12:51.855316 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 8808b44fbad7430b99d4003405af2bec
I20240503 02:12:51.859184 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 8808b44fbad7430b99d4003405af2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:51.859541 13394 tablet_replica.cc:330] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:51.859686 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb944e086c1f4287a1d4b5ec7aa87988" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:51.859844 13394 raft_consensus.cc:2229] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.859953 13040 raft_consensus.cc:2454] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:51.860136 13394 raft_consensus.cc:2258] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.860520 10427 leader_election.cc:302] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:51.861125 13394 ts_tablet_manager.cc:1875] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.861727 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 63e002f92dd64155bf425c9aaaa314ab
I20240503 02:12:51.862118 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 8808b44fbad7430b99d4003405af2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:51.862485 13397 tablet_replica.cc:330] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:51.862807 13397 raft_consensus.cc:2229] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:51.863268 13397 raft_consensus.cc:2258] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.864024 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e002f92dd64155bf425c9aaaa314ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:51.864298 13397 ts_tablet_manager.cc:1875] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.864703 13487 raft_consensus.cc:2781] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:51.864686 13384 raft_consensus.cc:482] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:51.864828 13487 raft_consensus.cc:482] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:51.864925 13487 raft_consensus.cc:3037] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.864867 13384 raft_consensus.cc:504] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.865975 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet b3a66c03cb7541b88142e0e93089d86c
I20240503 02:12:51.867902 13487 raft_consensus.cc:504] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.868083 13496 tablet_replica.cc:330] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:51.868422 13496 raft_consensus.cc:2229] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.868609 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a66c03cb7541b88142e0e93089d86c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:51.868785 13496 raft_consensus.cc:2258] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.869495 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e002f92dd64155bf425c9aaaa314ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:51.869757 13496 ts_tablet_manager.cc:1875] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.870728 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb944e086c1f4287a1d4b5ec7aa87988" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:51.870982 13040 raft_consensus.cc:3037] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.873426 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet 8808b44fbad7430b99d4003405af2bec with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:51.873750 13395 tablet_replica.cc:330] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:51.873780 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 63e002f92dd64155bf425c9aaaa314ab with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:51.874058 13395 raft_consensus.cc:2229] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.874122 13040 raft_consensus.cc:2454] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:51.874125 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a66c03cb7541b88142e0e93089d86c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:51.874409 13395 raft_consensus.cc:2258] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.874703 13487 leader_election.cc:288] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:51.874866 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet b3a66c03cb7541b88142e0e93089d86c with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:51 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:51.875329 13395 ts_tablet_manager.cc:1875] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.882928 13394 ts_tablet_manager.cc:1888] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.889127 13397 ts_tablet_manager.cc:1888] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.889298 13397 log.cc:1196] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/8808b44fbad7430b99d4003405af2bec
I20240503 02:12:51.892113 13394 log.cc:1196] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/8808b44fbad7430b99d4003405af2bec
I20240503 02:12:51.893244 13394 ts_tablet_manager.cc:1909] T 8808b44fbad7430b99d4003405af2bec P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:51.895077 13397 ts_tablet_manager.cc:1909] T 8808b44fbad7430b99d4003405af2bec P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:51.895736 13395 ts_tablet_manager.cc:1888] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.896037 13496 ts_tablet_manager.cc:1888] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.896220 13496 log.cc:1196] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/63e002f92dd64155bf425c9aaaa314ab
I20240503 02:12:51.896996 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 8808b44fbad7430b99d4003405af2bec (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:51.897472 13397 tablet_replica.cc:330] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:51.897797 13397 raft_consensus.cc:2229] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:51.898214 13397 raft_consensus.cc:2258] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.898741 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 8808b44fbad7430b99d4003405af2bec (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:51.895912 13395 log.cc:1196] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/8808b44fbad7430b99d4003405af2bec
I20240503 02:12:51.900180 13395 ts_tablet_manager.cc:1909] T 8808b44fbad7430b99d4003405af2bec P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:51.901340 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 8808b44fbad7430b99d4003405af2bec (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:51.901504 13394 tablet_replica.cc:330] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:51.900205 13397 ts_tablet_manager.cc:1875] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.901749 13394 raft_consensus.cc:2229] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.902041 13394 raft_consensus.cc:2258] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.902808 13394 ts_tablet_manager.cc:1875] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.903189 13384 leader_election.cc:288] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:51.904570 13496 ts_tablet_manager.cc:1909] T 63e002f92dd64155bf425c9aaaa314ab P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:51.905547 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be061441154fbba1e227d32474cd5e" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:51.909422 10427 leader_election.cc:302] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:51.909806 13487 raft_consensus.cc:2781] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:51.910007 13487 raft_consensus.cc:686] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:51.910423 13487 consensus_queue.cc:237] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:51.912631 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 63e002f92dd64155bf425c9aaaa314ab (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:51.923131 13394 ts_tablet_manager.cc:1888] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.923336 13394 log.cc:1196] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/b3a66c03cb7541b88142e0e93089d86c
I20240503 02:12:51.932788 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb944e086c1f4287a1d4b5ec7aa87988" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:51.933132 13173 raft_consensus.cc:2454] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:51.933893 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb944e086c1f4287a1d4b5ec7aa87988" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:51.934099 13173 raft_consensus.cc:3037] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.938076 13173 raft_consensus.cc:2454] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:51.939373 13310 raft_consensus.cc:1261] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:51.940002 12907 raft_consensus.cc:1261] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:51.940965 13429 consensus_queue.cc:1035] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.941157 13394 ts_tablet_manager.cc:1909] T b3a66c03cb7541b88142e0e93089d86c P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:51.943819 13394 tablet_replica.cc:330] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:51.944737 13394 raft_consensus.cc:2229] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:51.945297 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet b3a66c03cb7541b88142e0e93089d86c (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:51.945749 13394 raft_consensus.cc:2258] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.945693 13429 consensus_queue.cc:1035] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:51.946813 13394 ts_tablet_manager.cc:1875] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.949611 13397 ts_tablet_manager.cc:1888] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.953904 13397 log.cc:1196] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/b3a66c03cb7541b88142e0e93089d86c
I20240503 02:12:51.955550 13397 ts_tablet_manager.cc:1909] T b3a66c03cb7541b88142e0e93089d86c P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:51.961158 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet b3a66c03cb7541b88142e0e93089d86c (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:51.963238 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "67be061441154fbba1e227d32474cd5e" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:51.963672 10488 raft_consensus.cc:2379] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f93b103af083471b8e8c0376e9148aad in current term 1: Already voted for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in this term.
I20240503 02:12:51.964387 13107 leader_election.cc:302] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:51.965148 13384 raft_consensus.cc:3037] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:51.968959 13384 raft_consensus.cc:2726] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:51.973682 9951 catalog_manager.cc:5484] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:51.974292 13395 tablet_replica.cc:330] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:51.976521 9944 catalog_manager.cc:5484] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:51.978267 13395 raft_consensus.cc:2229] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:51.979583 13395 raft_consensus.cc:2258] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:51.980379 13395 ts_tablet_manager.cc:1875] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:51.990772 13394 ts_tablet_manager.cc:1888] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:51.990978 13394 log.cc:1196] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/63e002f92dd64155bf425c9aaaa314ab
I20240503 02:12:51.996855 13394 ts_tablet_manager.cc:1909] T 63e002f92dd64155bf425c9aaaa314ab P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:51.999737 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 63e002f92dd64155bf425c9aaaa314ab (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:52.006531 13395 ts_tablet_manager.cc:1888] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.006700 13395 log.cc:1196] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/63e002f92dd64155bf425c9aaaa314ab
I20240503 02:12:52.007150 13395 ts_tablet_manager.cc:1909] T 63e002f92dd64155bf425c9aaaa314ab P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:52.008383 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 63e002f92dd64155bf425c9aaaa314ab (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:52.015966 13395 tablet_replica.cc:330] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:52.016299 13395 raft_consensus.cc:2229] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.016608 13395 raft_consensus.cc:2258] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.019351 13395 ts_tablet_manager.cc:1875] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.030647 9944 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 23825a55f02c4e0a8bbcd07b147f626d (table rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd]). Expected version 1 got 0
I20240503 02:12:52.032186 9951 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 23825a55f02c4e0a8bbcd07b147f626d (table rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd]). Expected version 1 got 0
I20240503 02:12:52.033057 13535 mvcc.cc:205] Tried to move back new op lower bound from 7023420915454595072 to 7023420914070982656. Current Snapshot: MvccSnapshot[applied={T|T < 7023420915454595072}]
I20240503 02:12:52.043004 13541 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:52.044872 13541 tablet.cc:1620] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:52.045524 13542 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:52.048084 9944 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 23825a55f02c4e0a8bbcd07b147f626d (table rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd]). Expected version 1 got 0
I20240503 02:12:52.048792 13542 tablet.cc:1620] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:52.050609 9951 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd] alter complete (version 1)
I20240503 02:12:52.054630 9951 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 23825a55f02c4e0a8bbcd07b147f626d (table rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd]). Expected version 1 got 0
I20240503 02:12:52.055346 13475 raft_consensus.cc:482] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:52.055564 13475 raft_consensus.cc:504] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:52.056017 13395 ts_tablet_manager.cc:1888] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.056293 13475 leader_election.cc:288] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:52.057188 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae0e1e196444f9d50420d0f799754" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:52.057497 13173 raft_consensus.cc:2454] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:52.058184 12843 leader_election.cc:302] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:52.058632 13475 raft_consensus.cc:2781] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:52.058737 13475 raft_consensus.cc:482] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:52.058928 13475 raft_consensus.cc:3037] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:52.061385 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae0e1e196444f9d50420d0f799754" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:52.061638 10488 raft_consensus.cc:2454] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:52.062924 13395 log.cc:1196] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/b3a66c03cb7541b88142e0e93089d86c
I20240503 02:12:52.063547 13395 ts_tablet_manager.cc:1909] T b3a66c03cb7541b88142e0e93089d86c P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:52.063916 13475 raft_consensus.cc:504] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:52.065084 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet b3a66c03cb7541b88142e0e93089d86c (table rebalance_test_table_6 [id=765a1c12d1bc40c789b8d9063d2b5fa8]) successfully deleted
I20240503 02:12:52.065829 13543 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:52.066087 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae0e1e196444f9d50420d0f799754" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:52.066366 13173 raft_consensus.cc:3037] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:52.072064 13173 raft_consensus.cc:2454] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:52.072674 12843 leader_election.cc:302] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:52.073055 13527 raft_consensus.cc:2781] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:52.074604 13527 raft_consensus.cc:686] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:12:52.074983 13527 consensus_queue.cc:237] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:52.076748 13475 leader_election.cc:288] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:52.077540 13543 tablet.cc:1620] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:52.081014 10488 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "559ae0e1e196444f9d50420d0f799754" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:52.081250 10488 raft_consensus.cc:3037] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:52.086938 10488 raft_consensus.cc:2454] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:52.094111 9951 catalog_manager.cc:5484] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:52.170864 9951 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:52.185181 13541 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:52.185869 13541 tablet.cc:1620] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:52.186512 13542 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:52.187160 13542 tablet.cc:1620] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:52.187831 13543 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:52.188406 13543 tablet.cc:1620] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:52.197479 9951 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=54841543178e4a2fbb1be4e8085893cd] alter complete (version 2)
I20240503 02:12:52.239024 13527 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:52.252660 13475 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:52.308653 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:52.319432 13487 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:52.332309 13541 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:52.339742 13543 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:52.349181 13542 tablet.cc:1722] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:52.352787 13532 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:52.373888 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 02:12:52.374245 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_8" } modify_external_catalogs: true
I20240503 02:12:52.388294 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 2a1681e90d92423a8fd4f5702cff768f
I20240503 02:12:52.382086 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=54841543178e4a2fbb1be4e8085893cd] alter complete (version 3)
I20240503 02:12:52.396553 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1681e90d92423a8fd4f5702cff768f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:52.396993 13496 tablet_replica.cc:330] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:52.397372 13496 raft_consensus.cc:2229] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.397701 13496 raft_consensus.cc:2258] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.398480 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet d0aabb6d297143e2a7bf935fd4a01e84
I20240503 02:12:52.398746 13496 ts_tablet_manager.cc:1875] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.399366 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet d0aabb6d297143e2a7bf935fd4a01e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:52.399806 13394 tablet_replica.cc:330] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:52.400051 13394 raft_consensus.cc:2229] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.400149 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1681e90d92423a8fd4f5702cff768f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:52.400408 13394 raft_consensus.cc:2258] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.400513 13395 tablet_replica.cc:330] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:52.400635 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 2a1681e90d92423a8fd4f5702cff768f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:52.400751 13395 raft_consensus.cc:2229] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:52.401206 13395 raft_consensus.cc:2258] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.401288 13394 ts_tablet_manager.cc:1875] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.401410 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet d0aabb6d297143e2a7bf935fd4a01e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:52.401641 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet d0aabb6d297143e2a7bf935fd4a01e84 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:52.402029 13395 ts_tablet_manager.cc:1875] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.400781 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 8d10cf9defbc44978489aa90f16db80b
I20240503 02:12:52.407528 13496 ts_tablet_manager.cc:1888] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.407683 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d10cf9defbc44978489aa90f16db80b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:52.408658 13397 tablet_replica.cc:330] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:52.409003 13397 raft_consensus.cc:2229] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.409210 13394 ts_tablet_manager.cc:1888] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.409346 13394 log.cc:1196] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/d0aabb6d297143e2a7bf935fd4a01e84
I20240503 02:12:52.409370 13397 raft_consensus.cc:2258] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.409849 13394 ts_tablet_manager.cc:1909] T d0aabb6d297143e2a7bf935fd4a01e84 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:52.410377 13397 ts_tablet_manager.cc:1875] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.408947 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d10cf9defbc44978489aa90f16db80b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:52.411182 13496 log.cc:1196] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/2a1681e90d92423a8fd4f5702cff768f
I20240503 02:12:52.411381 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 8d10cf9defbc44978489aa90f16db80b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:52.411662 13496 ts_tablet_manager.cc:1909] T 2a1681e90d92423a8fd4f5702cff768f P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:52.411715 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet d0aabb6d297143e2a7bf935fd4a01e84 (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.412070 13394 tablet_replica.cc:330] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:52.412329 13394 raft_consensus.cc:2229] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.412847 13496 tablet_replica.cc:330] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:52.412720 13394 raft_consensus.cc:2258] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.413098 13496 raft_consensus.cc:2229] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.413367 13496 raft_consensus.cc:2258] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.413123 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 2a1681e90d92423a8fd4f5702cff768f (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.413884 13394 ts_tablet_manager.cc:1875] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.420135 13397 ts_tablet_manager.cc:1888] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.424858 13395 ts_tablet_manager.cc:1888] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.425158 13395 log.cc:1196] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/2a1681e90d92423a8fd4f5702cff768f
I20240503 02:12:52.425892 13397 log.cc:1196] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/2a1681e90d92423a8fd4f5702cff768f
I20240503 02:12:52.426230 13395 ts_tablet_manager.cc:1909] T 2a1681e90d92423a8fd4f5702cff768f P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:52.426770 13397 ts_tablet_manager.cc:1909] T 2a1681e90d92423a8fd4f5702cff768f P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:52.414283 13496 ts_tablet_manager.cc:1875] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.431696 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 2a1681e90d92423a8fd4f5702cff768f (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.432045 13394 ts_tablet_manager.cc:1888] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.432889 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 2a1681e90d92423a8fd4f5702cff768f (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.432267 13394 log.cc:1196] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/8d10cf9defbc44978489aa90f16db80b
I20240503 02:12:52.433352 13395 tablet_replica.cc:330] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:52.434688 13397 tablet_replica.cc:330] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:52.435003 13397 raft_consensus.cc:2229] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.435195 13395 raft_consensus.cc:2229] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:52.435500 13394 ts_tablet_manager.cc:1909] T 8d10cf9defbc44978489aa90f16db80b P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:52.436997 13397 raft_consensus.cc:2258] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.437237 13395 raft_consensus.cc:2258] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.437338 13496 ts_tablet_manager.cc:1888] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.437991 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 8d10cf9defbc44978489aa90f16db80b (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.438771 13397 ts_tablet_manager.cc:1875] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.442004 13395 ts_tablet_manager.cc:1875] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.444599 13496 log.cc:1196] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/8d10cf9defbc44978489aa90f16db80b
I20240503 02:12:52.445246 13496 ts_tablet_manager.cc:1909] T 8d10cf9defbc44978489aa90f16db80b P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:52.451144 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 8d10cf9defbc44978489aa90f16db80b (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.466781 13397 ts_tablet_manager.cc:1888] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.466985 13397 log.cc:1196] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/d0aabb6d297143e2a7bf935fd4a01e84
I20240503 02:12:52.468353 13395 ts_tablet_manager.cc:1888] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.468961 13397 ts_tablet_manager.cc:1909] T d0aabb6d297143e2a7bf935fd4a01e84 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:52.470297 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet d0aabb6d297143e2a7bf935fd4a01e84 (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.470606 13395 log.cc:1196] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/d0aabb6d297143e2a7bf935fd4a01e84
I20240503 02:12:52.471308 13395 ts_tablet_manager.cc:1909] T d0aabb6d297143e2a7bf935fd4a01e84 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:52.472712 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet d0aabb6d297143e2a7bf935fd4a01e84 (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.474444 13475 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:52.474974 13395 tablet_replica.cc:330] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:52.475281 13395 raft_consensus.cc:2229] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:52.475735 13395 raft_consensus.cc:2258] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.476687 13395 ts_tablet_manager.cc:1875] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.498319 13395 ts_tablet_manager.cc:1888] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.498584 13395 log.cc:1196] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/8d10cf9defbc44978489aa90f16db80b
I20240503 02:12:52.499182 13395 ts_tablet_manager.cc:1909] T 8d10cf9defbc44978489aa90f16db80b P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:52.505662 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 8d10cf9defbc44978489aa90f16db80b (table rebalance_test_table_8 [id=cd8b52d15f17414ca71b1fc43305acf4]) successfully deleted
I20240503 02:12:52.537683 13527 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:52.539093 9946 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:52.539356 9946 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:52.544696 9946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 23825a55f02c4e0a8bbcd07b147f626d
I20240503 02:12:52.545966 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 23825a55f02c4e0a8bbcd07b147f626d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:12:52.546006 12886 tablet_service.cc:1514] Processing DeleteTablet for tablet 23825a55f02c4e0a8bbcd07b147f626d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:12:52.547178 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 23825a55f02c4e0a8bbcd07b147f626d with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:12:52.560860 13577 tablet_replica.cc:330] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:12:52.561662 13577 raft_consensus.cc:2229] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.562443 13577 raft_consensus.cc:2258] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.565315 13577 ts_tablet_manager.cc:1875] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.568383 13579 tablet_replica.cc:330] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:12:52.568933 13579 raft_consensus.cc:2229] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.569370 13579 raft_consensus.cc:2258] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.574858 13577 ts_tablet_manager.cc:1888] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:52.575119 13577 log.cc:1196] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/23825a55f02c4e0a8bbcd07b147f626d
I20240503 02:12:52.575827 13577 ts_tablet_manager.cc:1909] T 23825a55f02c4e0a8bbcd07b147f626d P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:12:52.576740 13576 tablet_replica.cc:330] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:12:52.577126 13576 raft_consensus.cc:2229] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:52.577236 13579 ts_tablet_manager.cc:1875] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.578023 13576 raft_consensus.cc:2258] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.579411 13576 ts_tablet_manager.cc:1875] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.584234 13579 ts_tablet_manager.cc:1888] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:52.587124 13579 log.cc:1196] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/23825a55f02c4e0a8bbcd07b147f626d
I20240503 02:12:52.587522 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 23825a55f02c4e0a8bbcd07b147f626d (table rebalancer_dynamic_table_new_name [id=54841543178e4a2fbb1be4e8085893cd]) successfully deleted
I20240503 02:12:52.588227 13576 ts_tablet_manager.cc:1888] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:52.588366 13576 log.cc:1196] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/23825a55f02c4e0a8bbcd07b147f626d
I20240503 02:12:52.588829 13576 ts_tablet_manager.cc:1909] T 23825a55f02c4e0a8bbcd07b147f626d P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:12:52.589768 13579 ts_tablet_manager.cc:1909] T 23825a55f02c4e0a8bbcd07b147f626d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:12:52.590121 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 23825a55f02c4e0a8bbcd07b147f626d (table rebalancer_dynamic_table_new_name [id=54841543178e4a2fbb1be4e8085893cd]) successfully deleted
I20240503 02:12:52.591218 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 23825a55f02c4e0a8bbcd07b147f626d (table rebalancer_dynamic_table_new_name [id=54841543178e4a2fbb1be4e8085893cd]) successfully deleted
I20240503 02:12:52.637594 9944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:52.650453 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:52.662678 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 1578111fde6e43fd853142a48eb34ef5 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.663367 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1578111fde6e43fd853142a48eb34ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.664956 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 1578111fde6e43fd853142a48eb34ef5 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.665494 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1578111fde6e43fd853142a48eb34ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.671639 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 1578111fde6e43fd853142a48eb34ef5 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.672326 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1578111fde6e43fd853142a48eb34ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.679374 12886 tablet_service.cc:1467] Processing CreateTablet for tablet ee427bc54a9a488787e61519bb00f152 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.679922 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee427bc54a9a488787e61519bb00f152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.694021 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 9abc528ca83640c3bf66022cf2c9b914 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.694782 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9abc528ca83640c3bf66022cf2c9b914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.695770 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 9abc528ca83640c3bf66022cf2c9b914 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.696316 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9abc528ca83640c3bf66022cf2c9b914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.698084 13020 tablet_service.cc:1467] Processing CreateTablet for tablet ee427bc54a9a488787e61519bb00f152 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.698776 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee427bc54a9a488787e61519bb00f152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.706357 13604 tablet_bootstrap.cc:492] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:52.709723 13153 tablet_service.cc:1467] Processing CreateTablet for tablet ee427bc54a9a488787e61519bb00f152 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.709846 13019 tablet_service.cc:1467] Processing CreateTablet for tablet 9abc528ca83640c3bf66022cf2c9b914 (DEFAULT_TABLE table=rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:52.710268 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee427bc54a9a488787e61519bb00f152. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.710476 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9abc528ca83640c3bf66022cf2c9b914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.713022 13606 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:52.715251 13605 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:52.719377 13606 tablet_bootstrap.cc:654] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.723520 12887 tablet_service.cc:1467] Processing CreateTablet for tablet dd269676454749abba159ad338705a15 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0]), partition=
I20240503 02:12:52.724043 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd269676454749abba159ad338705a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.725407 13604 tablet_bootstrap.cc:654] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.726045 13608 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:52.726977 13602 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:52.729336 13608 tablet_bootstrap.cc:654] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.731576 13154 tablet_service.cc:1467] Processing CreateTablet for tablet dd269676454749abba159ad338705a15 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0]), partition=
I20240503 02:12:52.732051 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd269676454749abba159ad338705a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.742606 13606 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:52.742995 13606 ts_tablet_manager.cc:1367] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.006s
I20240503 02:12:52.745524 13603 tablet_bootstrap.cc:492] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:52.751019 13606 raft_consensus.cc:348] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:52.752099 13603 tablet_bootstrap.cc:654] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.753038 13604 tablet_bootstrap.cc:492] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:52.753135 13606 raft_consensus.cc:374] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.754019 13606 raft_consensus.cc:729] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.753906 13604 ts_tablet_manager.cc:1367] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.011s
I20240503 02:12:52.754343 13606 consensus_queue.cc:260] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:52.755601 13604 raft_consensus.cc:348] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:52.756491 13606 ts_tablet_manager.cc:1398] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20240503 02:12:52.756835 13604 raft_consensus.cc:374] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.757189 13604 raft_consensus.cc:729] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.757927 13604 consensus_queue.cc:260] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:52.759986 13604 ts_tablet_manager.cc:1398] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:52.760885 13604 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:52.763886 13604 tablet_bootstrap.cc:654] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.776027 13610 tablet_bootstrap.cc:492] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:52.779794 13604 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:52.780690 13604 ts_tablet_manager.cc:1367] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.007s
I20240503 02:12:52.782117 13604 raft_consensus.cc:348] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:52.783872 13604 raft_consensus.cc:374] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.784175 13604 raft_consensus.cc:729] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.784986 13604 consensus_queue.cc:260] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:52.787456 13604 ts_tablet_manager.cc:1398] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240503 02:12:52.821054 13288 tablet_service.cc:1467] Processing CreateTablet for tablet dd269676454749abba159ad338705a15 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0]), partition=
I20240503 02:12:52.821750 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd269676454749abba159ad338705a15. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:52.842694 13602 tablet_bootstrap.cc:654] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.856563 13610 tablet_bootstrap.cc:654] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.868875 13605 tablet_bootstrap.cc:654] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:52.915680 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 02:12:52.916079 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_10" } modify_external_catalogs: true
I20240503 02:12:52.922370 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 92c8ac796c0441199e0a47a8e4d3e1b0
I20240503 02:12:52.923055 13378 raft_consensus.cc:482] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:52.923295 13378 raft_consensus.cc:504] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:52.924710 13378 leader_election.cc:288] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:52.939154 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 2fdc2a02fa0d42eb93bbbe6213b4723a
I20240503 02:12:52.940593 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c8ac796c0441199e0a47a8e4d3e1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:52.941000 13397 tablet_replica.cc:330] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:52.941356 13397 raft_consensus.cc:2229] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:52.941886 13397 raft_consensus.cc:2258] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.941834 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c8ac796c0441199e0a47a8e4d3e1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:52.943084 13397 ts_tablet_manager.cc:1875] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.943341 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:52.943656 13496 tablet_replica.cc:330] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:52.943919 13496 raft_consensus.cc:2229] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:52.944315 13496 raft_consensus.cc:2258] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.944929 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:52.945297 13496 ts_tablet_manager.cc:1875] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.945297 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 94d6673d883d4e788ad3e7fe020d7713
I20240503 02:12:52.946166 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d6673d883d4e788ad3e7fe020d7713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:52.950493 13397 ts_tablet_manager.cc:1888] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.950662 13397 log.cc:1196] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/92c8ac796c0441199e0a47a8e4d3e1b0
I20240503 02:12:52.951179 13397 ts_tablet_manager.cc:1909] T 92c8ac796c0441199e0a47a8e4d3e1b0 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:52.952582 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 92c8ac796c0441199e0a47a8e4d3e1b0 (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:52.953506 13496 ts_tablet_manager.cc:1888] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.953773 13496 log.cc:1196] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/2fdc2a02fa0d42eb93bbbe6213b4723a
I20240503 02:12:52.953768 13619 tablet_replica.cc:330] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:52.954062 13619 raft_consensus.cc:2229] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.954401 13496 ts_tablet_manager.cc:1909] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:52.954389 13619 raft_consensus.cc:2258] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.955191 13619 ts_tablet_manager.cc:1875] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.955964 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 2fdc2a02fa0d42eb93bbbe6213b4723a (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:52.957836 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d6673d883d4e788ad3e7fe020d7713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:52.958259 13397 tablet_replica.cc:330] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:52.958612 13397 raft_consensus.cc:2229] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.959005 13397 raft_consensus.cc:2258] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.960523 13397 ts_tablet_manager.cc:1875] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:52.966437 13429 raft_consensus.cc:482] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:52.966650 13429 raft_consensus.cc:504] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:52.967738 13429 leader_election.cc:288] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:52.968969 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee427bc54a9a488787e61519bb00f152" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
W20240503 02:12:52.970168 12973 leader_election.cc:341] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:52.982714 10141 tablet.cc:2369] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:52.986411 13602 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:52.987368 13397 ts_tablet_manager.cc:1888] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.986960 13602 ts_tablet_manager.cc:1367] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.260s user 0.000s sys 0.007s
I20240503 02:12:52.988946 13602 raft_consensus.cc:348] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:52.990852 13619 ts_tablet_manager.cc:1888] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:52.991154 13608 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:52.991192 13619 log.cc:1196] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/92c8ac796c0441199e0a47a8e4d3e1b0
I20240503 02:12:52.992049 13619 ts_tablet_manager.cc:1909] T 92c8ac796c0441199e0a47a8e4d3e1b0 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:52.991361 13608 ts_tablet_manager.cc:1367] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.265s user 0.074s sys 0.074s
I20240503 02:12:52.993968 13610 tablet_bootstrap.cc:492] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:52.994982 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 92c8ac796c0441199e0a47a8e4d3e1b0 (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:52.995085 13605 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:52.995309 13605 ts_tablet_manager.cc:1367] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.280s user 0.004s sys 0.000s
I20240503 02:12:52.995599 13610 ts_tablet_manager.cc:1367] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.220s user 0.002s sys 0.003s
I20240503 02:12:52.995579 13608 raft_consensus.cc:348] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:52.996315 13608 raft_consensus.cc:374] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.996206 13619 tablet_replica.cc:330] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:52.996868 13619 raft_consensus.cc:2229] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:52.996904 13605 raft_consensus.cc:348] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:52.997035 13608 raft_consensus.cc:729] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.997769 13608 consensus_queue.cc:260] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:52.998260 13619 raft_consensus.cc:2258] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:52.998505 13610 raft_consensus.cc:348] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:52.998957 13605 raft_consensus.cc:374] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.999114 13610 raft_consensus.cc:374] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:52.999377 13610 raft_consensus.cc:729] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.999562 13605 raft_consensus.cc:729] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:52.999960 13610 consensus_queue.cc:260] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.000591 13605 consensus_queue.cc:260] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.002377 13487 raft_consensus.cc:482] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:53.002571 13487 raft_consensus.cc:504] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:53.002861 13610 ts_tablet_manager.cc:1398] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:12:53.003190 13610 tablet_bootstrap.cc:492] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:53.003734 13487 leader_election.cc:288] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:53.006363 13605 ts_tablet_manager.cc:1398] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:12:53.008286 13605 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:53.010879 13605 tablet_bootstrap.cc:654] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.015359 13610 tablet_bootstrap.cc:654] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.020601 13619 ts_tablet_manager.cc:1875] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.021824 13605 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:53.021198 13608 ts_tablet_manager.cc:1398] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20240503 02:12:53.023746 13610 tablet_bootstrap.cc:492] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:53.024622 13605 ts_tablet_manager.cc:1367] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20240503 02:12:53.025982 13605 raft_consensus.cc:348] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:53.026554 13610 ts_tablet_manager.cc:1367] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.002s
I20240503 02:12:53.028097 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 92c8ac796c0441199e0a47a8e4d3e1b0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:53.028329 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet 94d6673d883d4e788ad3e7fe020d7713 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:53.028566 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fdc2a02fa0d42eb93bbbe6213b4723a with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:52 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:53.041069 13628 tablet_replica.cc:330] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:53.041595 13628 raft_consensus.cc:2229] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.039561 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1578111fde6e43fd853142a48eb34ef5" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:53.027909 13610 raft_consensus.cc:348] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.042497 13605 raft_consensus.cc:374] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.042981 13628 raft_consensus.cc:2258] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.043183 13605 raft_consensus.cc:729] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.043408 13610 raft_consensus.cc:374] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.043838 13605 consensus_queue.cc:260] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:53.044283 13610 raft_consensus.cc:729] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.045228 13610 consensus_queue.cc:260] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.045642 13309 raft_consensus.cc:2454] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:53.046775 10424 leader_election.cc:302] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:53.047334 13532 raft_consensus.cc:2781] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:53.047485 13532 raft_consensus.cc:482] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:53.047609 13532 raft_consensus.cc:3037] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:52.987627 13397 log.cc:1196] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/94d6673d883d4e788ad3e7fe020d7713
I20240503 02:12:53.006284 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1578111fde6e43fd853142a48eb34ef5" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:53.048843 13397 ts_tablet_manager.cc:1909] T 94d6673d883d4e788ad3e7fe020d7713 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:53.049098 13605 ts_tablet_manager.cc:1398] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240503 02:12:53.049417 13605 tablet_bootstrap.cc:492] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:53.051383 13532 raft_consensus.cc:504] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:53.052000 13605 tablet_bootstrap.cc:654] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.052479 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 94d6673d883d4e788ad3e7fe020d7713 (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:53.053129 13610 ts_tablet_manager.cc:1398] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20240503 02:12:53.053535 13628 ts_tablet_manager.cc:1875] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.057240 13602 raft_consensus.cc:374] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.057435 13602 raft_consensus.cc:729] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.057895 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1578111fde6e43fd853142a48eb34ef5" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:53.058466 13602 consensus_queue.cc:260] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:53.058729 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1578111fde6e43fd853142a48eb34ef5" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:53.058888 13532 leader_election.cc:288] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:53.058969 13309 raft_consensus.cc:3037] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.058955 10226 raft_consensus.cc:2454] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:53.060065 13602 ts_tablet_manager.cc:1398] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.072s user 0.002s sys 0.000s
I20240503 02:12:53.061914 13309 raft_consensus.cc:2454] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:53.063436 10424 leader_election.cc:302] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters: eff8e8d11a4447218741bb994aa41082
I20240503 02:12:53.064594 13487 raft_consensus.cc:2781] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:53.065461 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9abc528ca83640c3bf66022cf2c9b914" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:53.066201 13309 raft_consensus.cc:2454] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:53.073899 13603 tablet_bootstrap.cc:492] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:53.076120 13603 ts_tablet_manager.cc:1367] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.331s user 0.075s sys 0.096s
I20240503 02:12:53.077332 13603 raft_consensus.cc:348] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.077965 13603 raft_consensus.cc:374] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.078132 13603 raft_consensus.cc:729] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.078122 13487 raft_consensus.cc:686] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:53.079490 13487 consensus_queue.cc:237] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:53.083415 10026 leader_election.cc:302] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:53.083801 13514 raft_consensus.cc:2781] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:53.083921 13514 raft_consensus.cc:482] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:53.084060 13514 raft_consensus.cc:3037] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.084749 13603 consensus_queue.cc:260] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.086246 13603 ts_tablet_manager.cc:1398] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:12:53.086565 13603 tablet_bootstrap.cc:492] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:53.086982 13514 raft_consensus.cc:504] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.087831 13514 leader_election.cc:288] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:53.089812 13603 tablet_bootstrap.cc:654] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.091086 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9abc528ca83640c3bf66022cf2c9b914" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:53.091393 13309 raft_consensus.cc:3037] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.094542 13309 raft_consensus.cc:2454] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:53.095535 10026 leader_election.cc:302] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:53.095175 9949 catalog_manager.cc:5484] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:53.097854 13514 raft_consensus.cc:2781] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:53.098145 13514 raft_consensus.cc:686] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:53.098486 13514 consensus_queue.cc:237] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.104498 13628 ts_tablet_manager.cc:1888] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.104535 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee427bc54a9a488787e61519bb00f152" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:53.104951 13173 raft_consensus.cc:2454] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:53.107197 13605 tablet_bootstrap.cc:492] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:53.107384 13605 ts_tablet_manager.cc:1367] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.000s
I20240503 02:12:53.108297 13605 raft_consensus.cc:348] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.109500 13605 raft_consensus.cc:374] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.109716 13605 raft_consensus.cc:729] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.110049 13605 consensus_queue.cc:260] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.110229 12976 leader_election.cc:302] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:53.110740 13605 ts_tablet_manager.cc:1398] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:53.110981 13429 raft_consensus.cc:2781] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:53.111116 13429 raft_consensus.cc:482] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:53.111213 13429 raft_consensus.cc:3037] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.117112 13429 raft_consensus.cc:504] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.118847 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee427bc54a9a488787e61519bb00f152" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:53.119129 12907 raft_consensus.cc:3037] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.120105 13429 leader_election.cc:288] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:53.121527 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee427bc54a9a488787e61519bb00f152" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:53.121778 13173 raft_consensus.cc:3037] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.122325 12907 raft_consensus.cc:2454] T ee427bc54a9a488787e61519bb00f152 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:53.122447 13628 log.cc:1196] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/92c8ac796c0441199e0a47a8e4d3e1b0
I20240503 02:12:53.122798 12973 leader_election.cc:302] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:53.123266 13429 raft_consensus.cc:2781] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:53.126252 13173 raft_consensus.cc:2454] T ee427bc54a9a488787e61519bb00f152 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:53.143450 13628 ts_tablet_manager.cc:1909] T 92c8ac796c0441199e0a47a8e4d3e1b0 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:53.144093 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9abc528ca83640c3bf66022cf2c9b914" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:53.144404 13040 raft_consensus.cc:2454] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:53.145308 13628 tablet_replica.cc:330] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:53.145414 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9abc528ca83640c3bf66022cf2c9b914" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:53.145627 13628 raft_consensus.cc:2229] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:53.145658 13040 raft_consensus.cc:3037] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.145978 13628 raft_consensus.cc:2258] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.146955 13628 ts_tablet_manager.cc:1875] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.149324 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 92c8ac796c0441199e0a47a8e4d3e1b0 (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:53.150314 13040 raft_consensus.cc:2454] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:53.151533 9951 catalog_manager.cc:5484] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:53.158784 13628 ts_tablet_manager.cc:1888] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.158988 13628 log.cc:1196] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/94d6673d883d4e788ad3e7fe020d7713
I20240503 02:12:53.159513 13628 ts_tablet_manager.cc:1909] T 94d6673d883d4e788ad3e7fe020d7713 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
W20240503 02:12:53.176672 10028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 147658us
I20240503 02:12:53.183234 13603 tablet_bootstrap.cc:492] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:53.183638 13603 ts_tablet_manager.cc:1367] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.097s user 0.012s sys 0.049s
I20240503 02:12:53.184666 13603 raft_consensus.cc:348] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.184818 13429 raft_consensus.cc:686] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:53.185053 13603 raft_consensus.cc:374] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.185140 13603 raft_consensus.cc:729] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.185428 13603 consensus_queue.cc:260] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.186058 13429 consensus_queue.cc:237] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.186533 13628 tablet_replica.cc:330] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:53.186585 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 94d6673d883d4e788ad3e7fe020d7713 (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:53.186980 13628 raft_consensus.cc:2229] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.187347 13628 raft_consensus.cc:2258] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.188750 13628 ts_tablet_manager.cc:1875] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.189432 13603 ts_tablet_manager.cc:1398] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:53.194358 13619 ts_tablet_manager.cc:1888] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.194531 13619 log.cc:1196] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/2fdc2a02fa0d42eb93bbbe6213b4723a
I20240503 02:12:53.194985 13619 ts_tablet_manager.cc:1909] T 2fdc2a02fa0d42eb93bbbe6213b4723a P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:53.196125 13619 tablet_replica.cc:330] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:53.196241 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 2fdc2a02fa0d42eb93bbbe6213b4723a (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:53.202093 13619 raft_consensus.cc:2229] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.203291 13619 raft_consensus.cc:2258] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.205277 13416 raft_consensus.cc:482] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:53.205559 13416 raft_consensus.cc:504] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.206593 13416 leader_election.cc:288] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:53.207233 13619 ts_tablet_manager.cc:1875] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.207628 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd269676454749abba159ad338705a15" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:53.207983 12907 raft_consensus.cc:2454] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:53.209122 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd269676454749abba159ad338705a15" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:53.209425 13173 raft_consensus.cc:2454] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:53.211457 9949 catalog_manager.cc:5484] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:53.216645 13243 leader_election.cc:302] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:53.217988 13416 raft_consensus.cc:2781] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:53.218122 13416 raft_consensus.cc:482] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:53.218253 13416 raft_consensus.cc:3037] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.227563 13416 raft_consensus.cc:504] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.228791 13619 ts_tablet_manager.cc:1888] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.229750 13619 log.cc:1196] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/94d6673d883d4e788ad3e7fe020d7713
I20240503 02:12:53.230553 13619 ts_tablet_manager.cc:1909] T 94d6673d883d4e788ad3e7fe020d7713 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:53.232681 13628 ts_tablet_manager.cc:1888] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.232856 13628 log.cc:1196] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/2fdc2a02fa0d42eb93bbbe6213b4723a
I20240503 02:12:53.234231 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 94d6673d883d4e788ad3e7fe020d7713 (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:53.234338 13628 ts_tablet_manager.cc:1909] T 2fdc2a02fa0d42eb93bbbe6213b4723a P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:53.237836 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd269676454749abba159ad338705a15" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:53.238128 12907 raft_consensus.cc:3037] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.241360 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 2fdc2a02fa0d42eb93bbbe6213b4723a (table rebalance_test_table_10 [id=06249801fa1c427d8f893d9fec4e975d]) successfully deleted
I20240503 02:12:53.243575 12907 raft_consensus.cc:2454] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:53.244959 13243 leader_election.cc:302] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:53.256582 13416 leader_election.cc:288] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:53.257143 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd269676454749abba159ad338705a15" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:53.257436 13173 raft_consensus.cc:3037] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.257551 13416 raft_consensus.cc:2781] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:53.257740 13416 raft_consensus.cc:686] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:12:53.259980 13173 raft_consensus.cc:2454] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:53.258118 13416 consensus_queue.cc:237] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:53.269073 9949 catalog_manager.cc:5484] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:53.285374 10073 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:12:53.286758 13020 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:53.298207 13288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
W20240503 02:12:53.305053 10274 tablet.cc:2369] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20240503 02:12:53.318707 12886 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:53.334970 10471 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:12:53.344004 10337 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:12:53.350112 13154 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:53.380700 10207 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:12:53.406145 9949 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:53.434412 13173 raft_consensus.cc:1261] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:53.435294 13648 consensus_queue.cc:1035] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:53.447022 12907 raft_consensus.cc:1261] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:53.451519 13648 consensus_queue.cc:1035] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:53.466217 9949 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 02:12:53.466445 9946 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet dd269676454749abba159ad338705a15 (table rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0]). Expected version 1 got 0
I20240503 02:12:53.466578 9949 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_12" } modify_external_catalogs: true
I20240503 02:12:53.473855 13660 mvcc.cc:205] Tried to move back new op lower bound from 7023420921574735872 to 7023420920881577984. Current Snapshot: MvccSnapshot[applied={T|T < 7023420921574735872}]
I20240503 02:12:53.479454 9949 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet b757782cdde440a58c2eafea93c7ad04
I20240503 02:12:53.482280 9949 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet ad56223da5874609a18d1b640c22e2f2
I20240503 02:12:53.482762 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet b757782cdde440a58c2eafea93c7ad04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:53.483347 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet b757782cdde440a58c2eafea93c7ad04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:53.483595 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet b757782cdde440a58c2eafea93c7ad04 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:53.483695 13397 tablet_replica.cc:330] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:53.483924 13397 raft_consensus.cc:2229] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:53.484522 13397 raft_consensus.cc:2258] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.484844 13628 tablet_replica.cc:330] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:53.485128 13628 raft_consensus.cc:2229] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.485422 13628 raft_consensus.cc:2258] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.485545 13397 ts_tablet_manager.cc:1875] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.486301 13628 ts_tablet_manager.cc:1875] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.498878 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet ad56223da5874609a18d1b640c22e2f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:53.499135 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet ad56223da5874609a18d1b640c22e2f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:53.502908 13628 ts_tablet_manager.cc:1888] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.503203 13628 log.cc:1196] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/b757782cdde440a58c2eafea93c7ad04
I20240503 02:12:53.503867 13628 ts_tablet_manager.cc:1909] T b757782cdde440a58c2eafea93c7ad04 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:53.504227 13397 ts_tablet_manager.cc:1888] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.504645 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet ad56223da5874609a18d1b640c22e2f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:53.504962 13619 tablet_replica.cc:330] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:53.505224 13628 tablet_replica.cc:330] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:53.505245 13619 raft_consensus.cc:2229] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.505432 13628 raft_consensus.cc:2229] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:53.505535 13619 raft_consensus.cc:2258] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.506433 13619 ts_tablet_manager.cc:1875] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.506915 13628 raft_consensus.cc:2258] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.507673 13628 ts_tablet_manager.cc:1875] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.508603 9949 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 408abb70150b4da3b1c5abab5e2a344b
I20240503 02:12:53.509773 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet b757782cdde440a58c2eafea93c7ad04 (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.509959 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 408abb70150b4da3b1c5abab5e2a344b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:53.511200 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet 408abb70150b4da3b1c5abab5e2a344b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:53.511612 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 408abb70150b4da3b1c5abab5e2a344b with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:53 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:53.516908 13628 ts_tablet_manager.cc:1888] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.517146 13628 log.cc:1196] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/ad56223da5874609a18d1b640c22e2f2
I20240503 02:12:53.517731 13628 ts_tablet_manager.cc:1909] T ad56223da5874609a18d1b640c22e2f2 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:53.518482 13619 ts_tablet_manager.cc:1888] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.518913 13628 tablet_replica.cc:330] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:53.519160 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet ad56223da5874609a18d1b640c22e2f2 (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.519248 13628 raft_consensus.cc:2229] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.518713 13619 log.cc:1196] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/ad56223da5874609a18d1b640c22e2f2
I20240503 02:12:53.519634 13628 raft_consensus.cc:2258] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.519919 13619 ts_tablet_manager.cc:1909] T ad56223da5874609a18d1b640c22e2f2 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:53.520531 13628 ts_tablet_manager.cc:1875] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.521822 13397 log.cc:1196] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/b757782cdde440a58c2eafea93c7ad04
I20240503 02:12:53.523085 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet ad56223da5874609a18d1b640c22e2f2 (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.524529 13397 ts_tablet_manager.cc:1909] T b757782cdde440a58c2eafea93c7ad04 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:53.524971 13619 tablet_replica.cc:330] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:53.526373 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet b757782cdde440a58c2eafea93c7ad04 (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.528530 13628 ts_tablet_manager.cc:1888] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.528833 13628 log.cc:1196] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/408abb70150b4da3b1c5abab5e2a344b
I20240503 02:12:53.528604 13619 raft_consensus.cc:2229] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:53.529841 13619 raft_consensus.cc:2258] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.530817 13636 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:53.533010 13665 tablet_replica.cc:330] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:53.537561 13397 tablet_replica.cc:330] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:53.538070 13397 raft_consensus.cc:2229] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.538409 13628 ts_tablet_manager.cc:1909] T 408abb70150b4da3b1c5abab5e2a344b P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:53.538805 13665 raft_consensus.cc:2229] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.539213 13665 raft_consensus.cc:2258] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.540071 13665 ts_tablet_manager.cc:1875] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.540804 13397 raft_consensus.cc:2258] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.541384 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 408abb70150b4da3b1c5abab5e2a344b (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.544678 9949 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet dd269676454749abba159ad338705a15 (table rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0]). Expected version 1 got 0
I20240503 02:12:53.546418 13619 ts_tablet_manager.cc:1875] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.552886 9946 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet dd269676454749abba159ad338705a15 (table rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0]). Expected version 1 got 0
I20240503 02:12:53.556103 13397 ts_tablet_manager.cc:1875] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.564357 13664 tablet.cc:1722] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:53.563000 13663 tablet.cc:1722] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:53.567137 13663 tablet.cc:1620] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:53.565197 13664 tablet.cc:1620] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:12:53.574398 13487 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:53.578315 13629 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:53.584039 13665 ts_tablet_manager.cc:1888] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.584785 13619 ts_tablet_manager.cc:1888] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.585031 13619 log.cc:1196] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/408abb70150b4da3b1c5abab5e2a344b
I20240503 02:12:53.585603 13619 ts_tablet_manager.cc:1909] T 408abb70150b4da3b1c5abab5e2a344b P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:53.586884 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 408abb70150b4da3b1c5abab5e2a344b (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.587359 13665 log.cc:1196] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/b757782cdde440a58c2eafea93c7ad04
I20240503 02:12:53.587970 13665 ts_tablet_manager.cc:1909] T b757782cdde440a58c2eafea93c7ad04 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:53.589473 13665 tablet_replica.cc:330] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:53.589570 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet b757782cdde440a58c2eafea93c7ad04 (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.589879 13665 raft_consensus.cc:2229] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:53.590276 13665 raft_consensus.cc:2258] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:53.591326 13665 ts_tablet_manager.cc:1875] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:53.600634 9946 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet dd269676454749abba159ad338705a15 (table rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0]). Expected version 1 got 0
I20240503 02:12:53.603204 13665 ts_tablet_manager.cc:1888] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.603471 13665 log.cc:1196] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/408abb70150b4da3b1c5abab5e2a344b
I20240503 02:12:53.603984 13665 ts_tablet_manager.cc:1909] T 408abb70150b4da3b1c5abab5e2a344b P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:53.605589 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 408abb70150b4da3b1c5abab5e2a344b (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.616789 13397 ts_tablet_manager.cc:1888] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:53.617148 13397 log.cc:1196] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/ad56223da5874609a18d1b640c22e2f2
I20240503 02:12:53.617817 13397 ts_tablet_manager.cc:1909] T ad56223da5874609a18d1b640c22e2f2 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:53.619480 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet ad56223da5874609a18d1b640c22e2f2 (table rebalance_test_table_12 [id=10c31ae7b8d143c492c4362cb8c0c027]) successfully deleted
I20240503 02:12:53.623498 9946 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet dd269676454749abba159ad338705a15 (table rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0]). Expected version 1 got 0
I20240503 02:12:53.626633 13673 tablet.cc:1722] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:53.627323 13673 tablet.cc:1620] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:53.630879 10226 raft_consensus.cc:3037] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.639655 13532 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:53.641124 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0] alter complete (version 1)
I20240503 02:12:53.658972 13429 consensus_queue.cc:1035] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:53.669378 13639 consensus_queue.cc:1035] T ee427bc54a9a488787e61519bb00f152 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:53.749810 9951 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:53.759992 13663 tablet.cc:1722] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:53.760659 13663 tablet.cc:1620] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:53.760995 13664 tablet.cc:1722] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:53.761559 13664 tablet.cc:1620] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:12:53.762816 13673 tablet.cc:1722] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:53.763443 13673 tablet.cc:1620] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:53.775014 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:53.782320 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=fe3a4d8b7827485787d91f7b7c26ade0] alter complete (version 2)
I20240503 02:12:53.789283 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 83f291bfdc064894ae0f7d6fc58655df (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.789520 13154 tablet_service.cc:1467] Processing CreateTablet for tablet 83f291bfdc064894ae0f7d6fc58655df (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.789799 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f291bfdc064894ae0f7d6fc58655df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.790040 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f291bfdc064894ae0f7d6fc58655df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.790406 10073 tablet_service.cc:1467] Processing CreateTablet for tablet 317ce62718cb412bbf04569cab14309f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.790882 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 317ce62718cb412bbf04569cab14309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.795346 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 83f291bfdc064894ae0f7d6fc58655df (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.795778 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f291bfdc064894ae0f7d6fc58655df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.799921 13019 tablet_service.cc:1467] Processing CreateTablet for tablet 317ce62718cb412bbf04569cab14309f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.800393 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 317ce62718cb412bbf04569cab14309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.805982 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 317ce62718cb412bbf04569cab14309f (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.806593 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 317ce62718cb412bbf04569cab14309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.807956 13697 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:53.811484 13697 tablet_bootstrap.cc:654] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.814216 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 432c8c5d42cb454fb7d1ca5af4782bb7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.814728 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432c8c5d42cb454fb7d1ca5af4782bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.814814 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 432c8c5d42cb454fb7d1ca5af4782bb7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.815336 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432c8c5d42cb454fb7d1ca5af4782bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.819622 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 432c8c5d42cb454fb7d1ca5af4782bb7 (DEFAULT_TABLE table=rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:53.820276 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432c8c5d42cb454fb7d1ca5af4782bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:53.824079 13698 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:53.831542 13698 tablet_bootstrap.cc:654] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.833372 13697 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:53.833770 13697 ts_tablet_manager.cc:1367] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.007s
I20240503 02:12:53.835067 13697 raft_consensus.cc:348] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:53.835558 13697 raft_consensus.cc:374] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.835713 13697 raft_consensus.cc:729] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.836159 13701 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:53.836136 13697 consensus_queue.cc:260] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:53.836977 13697 ts_tablet_manager.cc:1398] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:53.838037 13697 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:53.838240 13701 tablet_bootstrap.cc:654] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.840026 13697 tablet_bootstrap.cc:654] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.849773 13697 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:53.850157 13697 ts_tablet_manager.cc:1367] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20240503 02:12:53.851140 13697 raft_consensus.cc:348] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.851579 13697 raft_consensus.cc:374] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.851742 13697 raft_consensus.cc:729] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.852113 13697 consensus_queue.cc:260] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.854270 13697 ts_tablet_manager.cc:1398] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:53.855607 13706 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:53.861528 13706 tablet_bootstrap.cc:654] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.861753 13701 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:53.862562 13701 ts_tablet_manager.cc:1367] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20240503 02:12:53.864923 13698 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:53.865243 13698 ts_tablet_manager.cc:1367] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.000s
I20240503 02:12:53.866294 13701 raft_consensus.cc:348] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.866341 13698 raft_consensus.cc:348] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:53.866549 13706 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:53.866636 13698 raft_consensus.cc:374] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.866719 13706 ts_tablet_manager.cc:1367] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.003s
I20240503 02:12:53.866753 13698 raft_consensus.cc:729] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.867027 13698 consensus_queue.cc:260] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:53.867381 13701 raft_consensus.cc:374] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.867625 13701 raft_consensus.cc:729] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.867944 13706 raft_consensus.cc:348] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.868011 13701 consensus_queue.cc:260] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.868291 13706 raft_consensus.cc:374] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.868613 13701 ts_tablet_manager.cc:1398] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:53.868708 13706 raft_consensus.cc:729] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.869042 13706 consensus_queue.cc:260] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.869868 13701 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:53.874259 13698 ts_tablet_manager.cc:1398] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:12:53.875059 13701 tablet_bootstrap.cc:654] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.880849 13698 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:53.881142 13706 ts_tablet_manager.cc:1398] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20240503 02:12:53.886200 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:53.889016 13701 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:53.889811 13701 ts_tablet_manager.cc:1367] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.006s
I20240503 02:12:53.889884 13706 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:53.892223 13706 tablet_bootstrap.cc:654] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.893297 13701 raft_consensus.cc:348] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.895010 13701 raft_consensus.cc:374] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.895113 13701 raft_consensus.cc:729] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.895541 13698 tablet_bootstrap.cc:654] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.895416 13701 consensus_queue.cc:260] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.896054 13701 ts_tablet_manager.cc:1398] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.006s user 0.000s sys 0.001s
I20240503 02:12:53.899950 13663 tablet.cc:1722] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:53.906455 13673 tablet.cc:1722] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:53.909071 13664 tablet.cc:1722] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:53.913859 13698 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:53.914110 13698 ts_tablet_manager.cc:1367] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.005s
I20240503 02:12:53.915377 13698 raft_consensus.cc:348] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.915730 13698 raft_consensus.cc:374] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.915819 13698 raft_consensus.cc:729] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.916093 13698 consensus_queue.cc:260] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.916786 13698 ts_tablet_manager.cc:1398] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:53.917116 13698 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:53.919186 13698 tablet_bootstrap.cc:654] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:53.926988 13706 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:53.927248 13706 ts_tablet_manager.cc:1367] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.002s
I20240503 02:12:53.928210 13706 raft_consensus.cc:348] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:53.928493 13706 raft_consensus.cc:374] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.928594 13706 raft_consensus.cc:729] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.928622 13636 raft_consensus.cc:482] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:53.928833 13636 raft_consensus.cc:504] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.928930 13706 consensus_queue.cc:260] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:53.929874 13636 leader_election.cc:288] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:53.931964 13639 raft_consensus.cc:482] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:53.932163 13639 raft_consensus.cc:504] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.933470 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432c8c5d42cb454fb7d1ca5af4782bb7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:53.933769 13310 raft_consensus.cc:2454] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:53.934073 13639 leader_election.cc:288] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:53.934343 10026 leader_election.cc:302] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:53.934736 13636 raft_consensus.cc:2781] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:53.934870 13636 raft_consensus.cc:482] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:53.934983 13636 raft_consensus.cc:3037] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.939049 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317ce62718cb412bbf04569cab14309f" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:53.939469 13174 raft_consensus.cc:2454] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:53.940117 12976 leader_election.cc:302] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:53.941707 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=fe3a4d8b7827485787d91f7b7c26ade0] alter complete (version 3)
I20240503 02:12:53.942287 13706 ts_tablet_manager.cc:1398] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.015s user 0.000s sys 0.003s
I20240503 02:12:53.945964 13698 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:53.946172 13698 ts_tablet_manager.cc:1367] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.005s
I20240503 02:12:53.946583 13639 raft_consensus.cc:2781] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:53.946692 13639 raft_consensus.cc:482] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:53.946801 13639 raft_consensus.cc:3037] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.948401 13698 raft_consensus.cc:348] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.949514 13698 raft_consensus.cc:374] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:53.949787 13698 raft_consensus.cc:729] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:53.949877 13639 raft_consensus.cc:504] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.950359 13698 consensus_queue.cc:260] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.950686 13639 leader_election.cc:288] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:53.951221 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317ce62718cb412bbf04569cab14309f" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:53.951500 13174 raft_consensus.cc:3037] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.952679 13698 ts_tablet_manager.cc:1398] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20240503 02:12:53.955075 13174 raft_consensus.cc:2454] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:53.956061 13636 raft_consensus.cc:504] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.956977 13636 leader_election.cc:288] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:53.957556 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432c8c5d42cb454fb7d1ca5af4782bb7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:53.957852 13310 raft_consensus.cc:3037] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:53.961611 13310 raft_consensus.cc:2454] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:53.957907 13384 raft_consensus.cc:482] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:53.961911 13384 raft_consensus.cc:504] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.962250 10026 leader_election.cc:302] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:53.963080 13384 leader_election.cc:288] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:53.964140 13629 raft_consensus.cc:2781] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:53.964226 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432c8c5d42cb454fb7d1ca5af4782bb7" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:53.964344 13629 raft_consensus.cc:686] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:53.964516 13309 raft_consensus.cc:2379] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f93b103af083471b8e8c0376e9148aad in current term 1: Already voted for candidate 031a0afb01b6469299fa829fa3467ef7 in this term.
I20240503 02:12:53.964733 13629 consensus_queue.cc:237] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:53.970458 12976 leader_election.cc:302] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:53.971400 13429 raft_consensus.cc:2781] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:53.971594 13429 raft_consensus.cc:686] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:53.971881 13429 consensus_queue.cc:237] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:53.978578 9946 catalog_manager.cc:5484] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:53.998668 9951 catalog_manager.cc:5484] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:54.001402 10092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317ce62718cb412bbf04569cab14309f" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:54.001720 10092 raft_consensus.cc:2454] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:54.002112 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "317ce62718cb412bbf04569cab14309f" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:54.002337 10094 raft_consensus.cc:3037] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.006687 10094 raft_consensus.cc:2454] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:54.013800 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 02:12:54.014125 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_14" } modify_external_catalogs: true
I20240503 02:12:54.016530 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432c8c5d42cb454fb7d1ca5af4782bb7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:54.016815 13174 raft_consensus.cc:2454] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:54.017136 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432c8c5d42cb454fb7d1ca5af4782bb7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:54.017362 13173 raft_consensus.cc:3037] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.018661 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 288181c4acf049d9ac832141162e59bc
I20240503 02:12:54.019675 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 288181c4acf049d9ac832141162e59bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:54.019951 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet e9e25c582fdb488f91083c6f5add142f
I20240503 02:12:54.020041 13628 tablet_replica.cc:330] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:54.020309 13628 raft_consensus.cc:2229] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.020460 13173 raft_consensus.cc:2454] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:54.020655 13628 raft_consensus.cc:2258] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.020867 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e25c582fdb488f91083c6f5add142f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:54.021322 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 288181c4acf049d9ac832141162e59bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:54.021461 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 41b4e413cfc2460d9c9c05649c5a1ac5
I20240503 02:12:54.021535 13628 ts_tablet_manager.cc:1875] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.021718 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 288181c4acf049d9ac832141162e59bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:54.022056 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b4e413cfc2460d9c9c05649c5a1ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:54.022090 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e25c582fdb488f91083c6f5add142f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:54.022455 13397 tablet_replica.cc:330] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:54.022706 13397 raft_consensus.cc:2229] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:54.023169 13397 raft_consensus.cc:2258] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.024065 13397 ts_tablet_manager.cc:1875] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.024820 13665 tablet_replica.cc:330] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:54.024968 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet e9e25c582fdb488f91083c6f5add142f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:54.025074 13665 raft_consensus.cc:2229] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.025305 13619 tablet_replica.cc:330] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:54.025372 13665 raft_consensus.cc:2258] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.025544 13619 raft_consensus.cc:2229] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.025815 13619 raft_consensus.cc:2258] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.026131 13665 ts_tablet_manager.cc:1875] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.026598 13619 ts_tablet_manager.cc:1875] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.028117 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b4e413cfc2460d9c9c05649c5a1ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:54.028415 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 41b4e413cfc2460d9c9c05649c5a1ac5 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:54.035544 13665 ts_tablet_manager.cc:1888] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.040180 13665 log.cc:1196] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/e9e25c582fdb488f91083c6f5add142f
I20240503 02:12:54.041055 13665 ts_tablet_manager.cc:1909] T e9e25c582fdb488f91083c6f5add142f P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:54.042627 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet e9e25c582fdb488f91083c6f5add142f (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.043426 13619 ts_tablet_manager.cc:1888] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.044862 13397 ts_tablet_manager.cc:1888] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.045709 13397 log.cc:1196] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/288181c4acf049d9ac832141162e59bc
I20240503 02:12:54.046996 13628 ts_tablet_manager.cc:1888] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.047811 13619 log.cc:1196] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/288181c4acf049d9ac832141162e59bc
I20240503 02:12:54.049872 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "432c8c5d42cb454fb7d1ca5af4782bb7" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:54.051796 13628 log.cc:1196] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/288181c4acf049d9ac832141162e59bc
I20240503 02:12:54.052153 13109 leader_election.cc:302] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:12:54.052594 13384 raft_consensus.cc:2726] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:54.053112 13619 ts_tablet_manager.cc:1909] T 288181c4acf049d9ac832141162e59bc P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:54.055109 13397 ts_tablet_manager.cc:1909] T 288181c4acf049d9ac832141162e59bc P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:54.056324 13397 tablet_replica.cc:330] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:54.056505 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 288181c4acf049d9ac832141162e59bc (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.056617 13397 raft_consensus.cc:2229] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.056871 13397 raft_consensus.cc:2258] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.057826 13397 ts_tablet_manager.cc:1875] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.060292 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:54.060618 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:54.065454 13397 ts_tablet_manager.cc:1888] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.065721 13397 log.cc:1196] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/41b4e413cfc2460d9c9c05649c5a1ac5
I20240503 02:12:54.066362 13397 ts_tablet_manager.cc:1909] T 41b4e413cfc2460d9c9c05649c5a1ac5 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:54.068318 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 41b4e413cfc2460d9c9c05649c5a1ac5 (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.068645 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet dd269676454749abba159ad338705a15
I20240503 02:12:54.069425 12886 tablet_service.cc:1514] Processing DeleteTablet for tablet dd269676454749abba159ad338705a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:12:54.073108 13628 ts_tablet_manager.cc:1909] T 288181c4acf049d9ac832141162e59bc P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:54.074280 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 288181c4acf049d9ac832141162e59bc (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.075206 13619 tablet_replica.cc:330] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:54.075325 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 288181c4acf049d9ac832141162e59bc (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.075618 13628 tablet_replica.cc:330] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:54.075938 13628 raft_consensus.cc:2229] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.076090 13619 raft_consensus.cc:2229] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:54.076730 13628 raft_consensus.cc:2258] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.076485 13619 raft_consensus.cc:2258] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.077580 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet dd269676454749abba159ad338705a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:12:54.078464 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet dd269676454749abba159ad338705a15 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:12:54.078938 13628 ts_tablet_manager.cc:1875] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.077890 13619 ts_tablet_manager.cc:1875] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.089730 13416 raft_consensus.cc:482] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:54.089931 13416 raft_consensus.cc:504] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.090884 13416 leader_election.cc:288] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:54.091603 13736 tablet_replica.cc:330] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:12:54.092077 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f291bfdc064894ae0f7d6fc58655df" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:54.092411 13040 raft_consensus.cc:2454] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:54.095548 13246 leader_election.cc:302] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:54.096026 13416 raft_consensus.cc:2781] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:54.096154 13416 raft_consensus.cc:482] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:54.096253 13416 raft_consensus.cc:3037] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.098309 13736 raft_consensus.cc:2229] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:54.098704 13736 raft_consensus.cc:2258] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.099041 13735 tablet_replica.cc:330] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:12:54.099328 13735 raft_consensus.cc:2229] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.099637 13735 raft_consensus.cc:2258] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.100615 13736 ts_tablet_manager.cc:1875] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.100862 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f291bfdc064894ae0f7d6fc58655df" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:54.101294 13173 raft_consensus.cc:2454] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:54.103667 13735 ts_tablet_manager.cc:1875] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.103790 13416 raft_consensus.cc:504] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.111840 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f291bfdc064894ae0f7d6fc58655df" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:54.112287 13173 raft_consensus.cc:3037] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.113926 13628 ts_tablet_manager.cc:1888] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.115095 13619 ts_tablet_manager.cc:1888] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.115393 13628 log.cc:1196] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/e9e25c582fdb488f91083c6f5add142f
I20240503 02:12:54.115971 13628 ts_tablet_manager.cc:1909] T e9e25c582fdb488f91083c6f5add142f P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:54.116575 13619 log.cc:1196] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/e9e25c582fdb488f91083c6f5add142f
I20240503 02:12:54.117223 13619 ts_tablet_manager.cc:1909] T e9e25c582fdb488f91083c6f5add142f P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:54.119691 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "83f291bfdc064894ae0f7d6fc58655df" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:54.120002 13040 raft_consensus.cc:3037] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.120038 13173 raft_consensus.cc:2454] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:54.120647 13246 leader_election.cc:302] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:54.121026 13672 raft_consensus.cc:2781] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:54.121186 13672 raft_consensus.cc:686] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:12:54.121510 13672 consensus_queue.cc:237] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.122345 13736 ts_tablet_manager.cc:1888] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:54.122478 13736 log.cc:1196] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/dd269676454749abba159ad338705a15
I20240503 02:12:54.123059 13736 ts_tablet_manager.cc:1909] T dd269676454749abba159ad338705a15 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:12:54.125051 13040 raft_consensus.cc:2454] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:54.125221 13735 ts_tablet_manager.cc:1888] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:54.125453 13735 log.cc:1196] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/dd269676454749abba159ad338705a15
I20240503 02:12:54.125864 13619 tablet_replica.cc:330] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:54.125952 13416 leader_election.cc:288] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:54.126014 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet e9e25c582fdb488f91083c6f5add142f (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.126171 13735 ts_tablet_manager.cc:1909] T dd269676454749abba159ad338705a15 P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:12:54.126541 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet e9e25c582fdb488f91083c6f5add142f (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.126981 13628 tablet_replica.cc:330] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:54.127291 13628 raft_consensus.cc:2229] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:54.127655 13619 raft_consensus.cc:2229] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.128360 13619 raft_consensus.cc:2258] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.128583 13628 raft_consensus.cc:2258] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.129962 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet dd269676454749abba159ad338705a15 (table rebalancer_dynamic_table_new_name [id=fe3a4d8b7827485787d91f7b7c26ade0]) successfully deleted
I20240503 02:12:54.130482 13628 ts_tablet_manager.cc:1875] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.130645 13619 ts_tablet_manager.cc:1875] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.131839 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet dd269676454749abba159ad338705a15 (table rebalancer_dynamic_table_new_name [id=fe3a4d8b7827485787d91f7b7c26ade0]) successfully deleted
I20240503 02:12:54.134512 9952 catalog_manager.cc:5484] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:54.138999 13628 ts_tablet_manager.cc:1888] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.139186 13628 log.cc:1196] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/41b4e413cfc2460d9c9c05649c5a1ac5
I20240503 02:12:54.139707 13628 ts_tablet_manager.cc:1909] T 41b4e413cfc2460d9c9c05649c5a1ac5 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:54.142171 13734 tablet_replica.cc:330] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:12:54.144779 13619 ts_tablet_manager.cc:1888] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.145083 13619 log.cc:1196] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/41b4e413cfc2460d9c9c05649c5a1ac5
I20240503 02:12:54.145769 13619 ts_tablet_manager.cc:1909] T 41b4e413cfc2460d9c9c05649c5a1ac5 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:54.142511 13734 raft_consensus.cc:2229] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.146138 13734 raft_consensus.cc:2258] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.146183 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 41b4e413cfc2460d9c9c05649c5a1ac5 (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.147256 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 41b4e413cfc2460d9c9c05649c5a1ac5 (table rebalance_test_table_14 [id=8fd6a7127d284a76a644b4e5a9771231]) successfully deleted
I20240503 02:12:54.149617 13734 ts_tablet_manager.cc:1875] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.159541 13734 ts_tablet_manager.cc:1888] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:54.159853 13734 log.cc:1196] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/dd269676454749abba159ad338705a15
I20240503 02:12:54.160499 13734 ts_tablet_manager.cc:1909] T dd269676454749abba159ad338705a15 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:12:54.162606 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet dd269676454749abba159ad338705a15 (table rebalancer_dynamic_table_new_name [id=fe3a4d8b7827485787d91f7b7c26ade0]) successfully deleted
I20240503 02:12:54.180233 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:54.190968 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 7f054f44932640bdab328226673614ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8]), partition=
I20240503 02:12:54.191519 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f054f44932640bdab328226673614ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.196874 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 7f054f44932640bdab328226673614ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8]), partition=
I20240503 02:12:54.197242 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f054f44932640bdab328226673614ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.201362 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 7f054f44932640bdab328226673614ac (DEFAULT_TABLE table=rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8]), partition=
I20240503 02:12:54.201862 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7f054f44932640bdab328226673614ac. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.234398 13698 tablet_bootstrap.cc:492] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:54.234532 10073 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:12:54.236850 13698 tablet_bootstrap.cc:654] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.239650 13697 tablet_bootstrap.cc:492] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:54.243014 13697 tablet_bootstrap.cc:654] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.262820 13761 tablet_bootstrap.cc:492] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:54.270002 13761 tablet_bootstrap.cc:654] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.277392 13697 tablet_bootstrap.cc:492] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:54.277732 13697 ts_tablet_manager.cc:1367] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.011s
I20240503 02:12:54.278841 13697 raft_consensus.cc:348] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.279285 13697 raft_consensus.cc:374] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.279458 13697 raft_consensus.cc:729] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.279872 13697 consensus_queue.cc:260] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.281806 13697 ts_tablet_manager.cc:1398] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:54.286042 13698 tablet_bootstrap.cc:492] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:54.287127 13698 ts_tablet_manager.cc:1367] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.053s user 0.001s sys 0.012s
I20240503 02:12:54.288951 13698 raft_consensus.cc:348] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.289323 13698 raft_consensus.cc:374] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.289463 13698 raft_consensus.cc:729] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.290131 13698 consensus_queue.cc:260] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.299221 10471 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:12:54.302183 10337 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:12:54.329455 13698 ts_tablet_manager.cc:1398] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.041s user 0.002s sys 0.000s
I20240503 02:12:54.329758 13153 tablet_service.cc:1429] Tablet server has 1 leaders and 0 scanners
I20240503 02:12:54.340101 13288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:54.343586 10207 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:12:54.345382 13416 raft_consensus.cc:482] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:54.345590 13416 raft_consensus.cc:504] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.346467 13416 leader_election.cc:288] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:54.347285 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f054f44932640bdab328226673614ac" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:54.347506 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f054f44932640bdab328226673614ac" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:54.347726 13173 raft_consensus.cc:2454] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
W20240503 02:12:54.348302 13243 leader_election.cc:341] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:54.349364 13246 leader_election.cc:302] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:54.351999 13416 raft_consensus.cc:2781] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:54.352160 13416 raft_consensus.cc:482] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:54.352257 13416 raft_consensus.cc:3037] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.355734 13416 raft_consensus.cc:504] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.360579 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f054f44932640bdab328226673614ac" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
W20240503 02:12:54.361852 13243 leader_election.cc:341] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:54.368161 13761 tablet_bootstrap.cc:492] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:54.368528 13761 ts_tablet_manager.cc:1367] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.106s user 0.026s sys 0.034s
I20240503 02:12:54.369626 13761 raft_consensus.cc:348] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.370007 13761 raft_consensus.cc:374] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.370159 13761 raft_consensus.cc:729] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.370627 13761 consensus_queue.cc:260] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.373085 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f054f44932640bdab328226673614ac" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:54.373364 13173 raft_consensus.cc:3037] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.374795 13761 ts_tablet_manager.cc:1398] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:54.376682 13416 leader_election.cc:288] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:54.377154 13173 raft_consensus.cc:2454] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:54.377960 13246 leader_election.cc:302] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:54.380138 13416 raft_consensus.cc:2781] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:54.380825 13416 raft_consensus.cc:686] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:12:54.381283 13416 consensus_queue.cc:237] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.396291 12886 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:54.402885 13019 tablet_service.cc:1429] Tablet server has 3 leaders and 0 scanners
I20240503 02:12:54.415050 13712 raft_consensus.cc:482] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:54.415309 13712 raft_consensus.cc:504] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:54.416213 13712 leader_election.cc:288] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:54.418574 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f054f44932640bdab328226673614ac" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:54.420329 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7f054f44932640bdab328226673614ac" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:54.420825 13173 raft_consensus.cc:2379] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in current term 1: Already voted for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in this term.
I20240503 02:12:54.421624 12843 leader_election.cc:302] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad
I20240503 02:12:54.422358 13712 raft_consensus.cc:3037] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.426127 13712 raft_consensus.cc:2726] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:54.437000 9949 catalog_manager.cc:5484] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:54.438498 13429 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:54.455273 13639 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:54.525200 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 02:12:54.525537 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_16" } modify_external_catalogs: true
I20240503 02:12:54.533144 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 6af8995538bb41f4bf7d2f7e0b5907d2
I20240503 02:12:54.533285 13416 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:54.534137 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af8995538bb41f4bf7d2f7e0b5907d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:54.534524 13628 tablet_replica.cc:330] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:54.534813 13628 raft_consensus.cc:2229] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.534909 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af8995538bb41f4bf7d2f7e0b5907d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:54.535080 13628 raft_consensus.cc:2258] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.535254 13397 tablet_replica.cc:330] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:54.535488 13397 raft_consensus.cc:2229] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:54.535900 13397 raft_consensus.cc:2258] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.536010 13628 ts_tablet_manager.cc:1875] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.536808 13397 ts_tablet_manager.cc:1875] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.542150 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 97432e625f864b91bcaabd8bf3400ff8
I20240503 02:12:54.542172 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 6af8995538bb41f4bf7d2f7e0b5907d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:54.542490 13665 tablet_replica.cc:330] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:54.542788 13665 raft_consensus.cc:2229] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.543081 13665 raft_consensus.cc:2258] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.543905 13665 ts_tablet_manager.cc:1875] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.544319 13628 ts_tablet_manager.cc:1888] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.546844 13628 log.cc:1196] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/6af8995538bb41f4bf7d2f7e0b5907d2
I20240503 02:12:54.547308 13628 ts_tablet_manager.cc:1909] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:54.548038 13636 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:54.550490 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 6af8995538bb41f4bf7d2f7e0b5907d2 (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.550781 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:54.571619 13636 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:54.572626 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 97432e625f864b91bcaabd8bf3400ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:54.572988 13619 tablet_replica.cc:330] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:54.573239 13619 raft_consensus.cc:2229] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.573499 13619 raft_consensus.cc:2258] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.574339 13619 ts_tablet_manager.cc:1875] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.585647 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet a9fb56b18eb14f588d261ee79ad97eea
I20240503 02:12:54.586596 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 97432e625f864b91bcaabd8bf3400ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:54.586565 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 97432e625f864b91bcaabd8bf3400ff8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:54.587007 13628 tablet_replica.cc:330] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:54.587335 13628 raft_consensus.cc:2229] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:54.587828 13628 raft_consensus.cc:2258] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.588407 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fb56b18eb14f588d261ee79ad97eea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:54.588744 13628 ts_tablet_manager.cc:1875] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.591531 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fb56b18eb14f588d261ee79ad97eea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:54.592958 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet a9fb56b18eb14f588d261ee79ad97eea with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:54 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:54.597635 13628 ts_tablet_manager.cc:1888] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.598476 13628 log.cc:1196] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/97432e625f864b91bcaabd8bf3400ff8
I20240503 02:12:54.599143 13628 ts_tablet_manager.cc:1909] T 97432e625f864b91bcaabd8bf3400ff8 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:54.602075 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 97432e625f864b91bcaabd8bf3400ff8 (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.605088 13628 tablet_replica.cc:330] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:54.605432 13628 raft_consensus.cc:2229] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.605789 13628 raft_consensus.cc:2258] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.606670 13628 ts_tablet_manager.cc:1875] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.607980 13665 ts_tablet_manager.cc:1888] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.611382 13619 ts_tablet_manager.cc:1888] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.620577 13628 ts_tablet_manager.cc:1888] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.622220 13397 ts_tablet_manager.cc:1888] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.622399 13397 log.cc:1196] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/6af8995538bb41f4bf7d2f7e0b5907d2
I20240503 02:12:54.622519 13628 log.cc:1196] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/a9fb56b18eb14f588d261ee79ad97eea
I20240503 02:12:54.624135 13628 ts_tablet_manager.cc:1909] T a9fb56b18eb14f588d261ee79ad97eea P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:54.626125 13665 log.cc:1196] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/6af8995538bb41f4bf7d2f7e0b5907d2
I20240503 02:12:54.626664 13665 ts_tablet_manager.cc:1909] T 6af8995538bb41f4bf7d2f7e0b5907d2 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:54.627599 13416 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:54.629062 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet a9fb56b18eb14f588d261ee79ad97eea (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.630158 13619 log.cc:1196] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/97432e625f864b91bcaabd8bf3400ff8
I20240503 02:12:54.630950 13619 ts_tablet_manager.cc:1909] T 97432e625f864b91bcaabd8bf3400ff8 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:54.634953 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 97432e625f864b91bcaabd8bf3400ff8 (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.635843 13397 ts_tablet_manager.cc:1909] T 6af8995538bb41f4bf7d2f7e0b5907d2 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:54.636613 13665 tablet_replica.cc:330] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:54.636732 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 6af8995538bb41f4bf7d2f7e0b5907d2 (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.639775 13619 tablet_replica.cc:330] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:54.640239 13665 raft_consensus.cc:2229] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.640641 13619 raft_consensus.cc:2229] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:54.640781 12908 raft_consensus.cc:1261] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:54.642107 13665 raft_consensus.cc:2258] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.642669 13416 consensus_queue.cc:1035] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:54.643543 13619 raft_consensus.cc:2258] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.645426 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:54.647847 13619 ts_tablet_manager.cc:1875] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.648311 13397 tablet_replica.cc:330] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:54.651789 13665 ts_tablet_manager.cc:1875] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.652696 13397 raft_consensus.cc:2229] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:54.653095 13397 raft_consensus.cc:2258] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:54.656080 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 6af8995538bb41f4bf7d2f7e0b5907d2 (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.656710 13397 ts_tablet_manager.cc:1875] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:54.666213 12886 tablet_service.cc:1467] Processing CreateTablet for tablet d9a312f6358e41d7abe66b2a6339e7f4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.666194 13019 tablet_service.cc:1467] Processing CreateTablet for tablet d9a312f6358e41d7abe66b2a6339e7f4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.666735 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a312f6358e41d7abe66b2a6339e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.666757 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a312f6358e41d7abe66b2a6339e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.668942 10471 tablet_service.cc:1467] Processing CreateTablet for tablet dd081871c430470b95fce8f19634dc2e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.669214 9952 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8]). Expected version 1 got 0
I20240503 02:12:54.669400 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd081871c430470b95fce8f19634dc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.672365 13288 tablet_service.cc:1467] Processing CreateTablet for tablet d9a312f6358e41d7abe66b2a6339e7f4 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.673228 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a312f6358e41d7abe66b2a6339e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.685724 13665 ts_tablet_manager.cc:1888] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.686481 12887 tablet_service.cc:1467] Processing CreateTablet for tablet dd081871c430470b95fce8f19634dc2e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.687052 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd081871c430470b95fce8f19634dc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.690191 13290 tablet_service.cc:1467] Processing CreateTablet for tablet dd081871c430470b95fce8f19634dc2e (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.682267 13619 ts_tablet_manager.cc:1888] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.690853 13619 log.cc:1196] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/a9fb56b18eb14f588d261ee79ad97eea
I20240503 02:12:54.690866 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd081871c430470b95fce8f19634dc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.691458 13619 ts_tablet_manager.cc:1909] T a9fb56b18eb14f588d261ee79ad97eea P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:54.691696 13665 log.cc:1196] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/a9fb56b18eb14f588d261ee79ad97eea
I20240503 02:12:54.692226 13665 ts_tablet_manager.cc:1909] T a9fb56b18eb14f588d261ee79ad97eea P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:54.693728 12888 tablet_service.cc:1467] Processing CreateTablet for tablet 09a142900b5c4909ad16b4abc70ee952 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.694219 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a142900b5c4909ad16b4abc70ee952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.694734 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 09a142900b5c4909ad16b4abc70ee952 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.695212 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a142900b5c4909ad16b4abc70ee952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.697971 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet a9fb56b18eb14f588d261ee79ad97eea (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.701933 13761 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:54.704419 13697 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:54.705045 13761 tablet_bootstrap.cc:654] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.706521 13697 tablet_bootstrap.cc:654] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.710978 13782 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:54.713142 13779 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:54.716285 13777 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:54.717064 13779 tablet_bootstrap.cc:654] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.717293 13777 tablet.cc:1620] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:54.719952 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet a9fb56b18eb14f588d261ee79ad97eea (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.721750 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 09a142900b5c4909ad16b4abc70ee952 (DEFAULT_TABLE table=rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:54.722193 13782 tablet_bootstrap.cc:654] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.722291 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a142900b5c4909ad16b4abc70ee952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:54.734295 9952 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8]). Expected version 1 got 0
I20240503 02:12:54.737998 13397 ts_tablet_manager.cc:1888] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:54.741283 13779 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:54.745443 13779 ts_tablet_manager.cc:1367] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.002s
I20240503 02:12:54.746404 13397 log.cc:1196] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/97432e625f864b91bcaabd8bf3400ff8
I20240503 02:12:54.747203 13779 raft_consensus.cc:348] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.747418 13397 ts_tablet_manager.cc:1909] T 97432e625f864b91bcaabd8bf3400ff8 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:54.748512 13790 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:54.748979 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 97432e625f864b91bcaabd8bf3400ff8 (table rebalance_test_table_16 [id=3c29f42af146408bb6a96abfd1cfef17]) successfully deleted
I20240503 02:12:54.753258 13779 raft_consensus.cc:374] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.755791 13790 tablet_bootstrap.cc:654] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.756053 13779 raft_consensus.cc:729] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.757218 13779 consensus_queue.cc:260] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.760022 13174 raft_consensus.cc:1261] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:54.760767 13779 ts_tablet_manager.cc:1398] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20240503 02:12:54.761155 13779 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:54.765507 13779 tablet_bootstrap.cc:654] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.767035 13778 consensus_queue.cc:1035] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:54.772678 13697 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:54.772998 13697 ts_tablet_manager.cc:1367] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.069s user 0.012s sys 0.009s
I20240503 02:12:54.773986 13697 raft_consensus.cc:348] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.776890 13697 raft_consensus.cc:374] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.778101 13697 raft_consensus.cc:729] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.779263 13697 consensus_queue.cc:260] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.781816 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8] alter complete (version 1)
I20240503 02:12:54.782377 13697 ts_tablet_manager.cc:1398] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:12:54.783655 13697 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:54.787984 13697 tablet_bootstrap.cc:654] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:54.790210 13790 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:54.790488 13790 ts_tablet_manager.cc:1367] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.000s
I20240503 02:12:54.791530 13790 raft_consensus.cc:348] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.791875 13790 raft_consensus.cc:374] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.792534 13790 raft_consensus.cc:729] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.792955 13790 consensus_queue.cc:260] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.795174 13790 ts_tablet_manager.cc:1398] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:54.793200 13776 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:54.797407 13776 tablet.cc:1620] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
W20240503 02:12:54.798408 10159 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.68:35025, user_credentials={real_user=slave}} blocked reactor thread for 79828.5us
I20240503 02:12:54.809782 13697 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:54.810525 13697 ts_tablet_manager.cc:1367] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.006s
I20240503 02:12:54.812964 13761 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:54.813205 13761 ts_tablet_manager.cc:1367] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.111s user 0.004s sys 0.007s
I20240503 02:12:54.814431 13761 raft_consensus.cc:348] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.814796 13761 raft_consensus.cc:374] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.814949 13761 raft_consensus.cc:729] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.815284 13761 consensus_queue.cc:260] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.816179 10357 consensus_queue.cc:237] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:12:54.816743 13697 raft_consensus.cc:348] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:54.817027 13697 raft_consensus.cc:374] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.817148 13697 raft_consensus.cc:729] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.818811 13697 consensus_queue.cc:260] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:54.820842 13782 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:54.821081 13782 ts_tablet_manager.cc:1367] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.111s user 0.028s sys 0.028s
I20240503 02:12:54.822083 13782 raft_consensus.cc:348] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:54.822407 13782 raft_consensus.cc:374] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:54.823521 13782 raft_consensus.cc:729] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:54.823570 13697 ts_tablet_manager.cc:1398] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:12:54.825057 10094 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:54.826368 12719 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:54.826582 10490 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:54.827224 12719 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:54.831856 13782 consensus_queue.cc:260] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:54.833302 13782 ts_tablet_manager.cc:1398] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
W20240503 02:12:54.848958 10295 consensus_peers.cc:485] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1fa4eb577d7f42fd8e4a9a4391a0546f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:54.888435 12719 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:12:54.889214 10489 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:12:54.889837 9952 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:54.906934 9944 catalog_manager.cc:5484] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:12:54.909862 13776 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:54.910490 13776 tablet.cc:1620] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:54.923840 13429 raft_consensus.cc:482] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:54.924084 13429 raft_consensus.cc:504] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.924955 13429 leader_election.cc:288] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:54.925179 9944 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8] alter complete (version 2)
I20240503 02:12:54.926208 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a312f6358e41d7abe66b2a6339e7f4" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:54.926231 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a312f6358e41d7abe66b2a6339e7f4" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:54.926489 13310 raft_consensus.cc:2454] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:54.926529 12907 raft_consensus.cc:2454] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:54.927150 12973 leader_election.cc:302] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:54.928679 13429 raft_consensus.cc:2781] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:54.928813 13429 raft_consensus.cc:482] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:54.928916 13429 raft_consensus.cc:3037] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.934289 13429 raft_consensus.cc:504] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.936328 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a312f6358e41d7abe66b2a6339e7f4" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:54.936780 12907 raft_consensus.cc:3037] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.937391 13375 raft_consensus.cc:482] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:54.937572 13375 raft_consensus.cc:504] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:54.938596 13375 leader_election.cc:288] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:54.943722 12907 raft_consensus.cc:2454] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:54.944665 12973 leader_election.cc:302] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:54.945132 13639 raft_consensus.cc:2781] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:54.949003 13777 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:54.949705 13777 tablet.cc:1620] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:54.950350 13429 leader_election.cc:288] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:54.952064 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d9a312f6358e41d7abe66b2a6339e7f4" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:54.952359 13310 raft_consensus.cc:3037] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:54.954566 13761 ts_tablet_manager.cc:1398] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.141s user 0.003s sys 0.000s
I20240503 02:12:54.955776 13310 raft_consensus.cc:2454] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:54.900694 10094 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:12:54.969843 13761 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:54.977095 13804 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:54.978004 13804 tablet.cc:1620] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:12:54.991077 13639 raft_consensus.cc:686] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:54.997154 13639 consensus_queue.cc:237] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.000344 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a142900b5c4909ad16b4abc70ee952" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:55.000741 13779 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:55.001098 13779 ts_tablet_manager.cc:1367] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.240s user 0.040s sys 0.021s
W20240503 02:12:55.001605 10159 leader_election.cc:341] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:55.002614 13779 raft_consensus.cc:348] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.003104 13779 raft_consensus.cc:374] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.003242 13779 raft_consensus.cc:729] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.004410 9952 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8]). Expected version 2 got 0
I20240503 02:12:55.009261 9946 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8]). Expected version 2 got 1
I20240503 02:12:55.011312 13786 raft_consensus.cc:482] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:55.011510 13786 raft_consensus.cc:504] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.013598 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd081871c430470b95fce8f19634dc2e" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:55.013967 13310 raft_consensus.cc:2454] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:55.014801 10424 leader_election.cc:302] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:55.015183 13781 raft_consensus.cc:2781] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:55.015319 13781 raft_consensus.cc:482] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:55.015484 13781 raft_consensus.cc:3037] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.016052 13761 tablet_bootstrap.cc:654] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.018213 13786 leader_election.cc:288] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:55.018826 13781 raft_consensus.cc:504] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.019688 13781 leader_election.cc:288] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:55.020864 9949 catalog_manager.cc:5484] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:55.022059 13779 consensus_queue.cc:260] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.024789 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a142900b5c4909ad16b4abc70ee952" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:55.025908 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd081871c430470b95fce8f19634dc2e" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:55.026199 13310 raft_consensus.cc:3037] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.032797 13310 raft_consensus.cc:2454] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:55.033643 10424 leader_election.cc:302] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:55.034163 13781 raft_consensus.cc:2781] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:55.034428 13781 raft_consensus.cc:686] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:55.034839 13781 consensus_queue.cc:237] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.036010 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd081871c430470b95fce8f19634dc2e" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:55.036093 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd081871c430470b95fce8f19634dc2e" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
W20240503 02:12:55.036667 10424 leader_election.cc:341] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:12:55.037077 10424 leader_election.cc:341] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:55.037501 13040 raft_consensus.cc:2454] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:55.038796 10162 leader_election.cc:302] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, eff8e8d11a4447218741bb994aa41082; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:55.040655 13779 ts_tablet_manager.cc:1398] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240503 02:12:55.044690 13375 raft_consensus.cc:2781] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:55.044872 13375 raft_consensus.cc:482] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:55.044922 9944 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table [id=b57f18a50b8548f79c96cda6440da2b8]). Expected version 2 got 1
I20240503 02:12:55.044987 13375 raft_consensus.cc:3037] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.049073 13375 raft_consensus.cc:504] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.049368 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:55.050379 13375 leader_election.cc:288] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:55.051059 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a142900b5c4909ad16b4abc70ee952" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
W20240503 02:12:55.052141 10159 leader_election.cc:341] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:55.054061 9952 catalog_manager.cc:5484] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:55.058604 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a142900b5c4909ad16b4abc70ee952" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:55.058933 13040 raft_consensus.cc:3037] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.062105 13040 raft_consensus.cc:2454] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:55.062930 10162 leader_election.cc:302] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, eff8e8d11a4447218741bb994aa41082; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:55.063879 13375 raft_consensus.cc:2781] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:55.064240 13375 raft_consensus.cc:686] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:55.064838 13375 consensus_queue.cc:237] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.066438 13776 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:55.067373 13777 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:55.084806 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=b57f18a50b8548f79c96cda6440da2b8] alter complete (version 3)
I20240503 02:12:55.089313 13804 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:55.090232 13804 tablet.cc:1620] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:12:55.090346 13761 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:55.090507 10489 consensus_queue.cc:237] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:12:55.090802 13761 ts_tablet_manager.cc:1367] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.121s user 0.005s sys 0.001s
I20240503 02:12:55.093158 9944 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 02:12:55.093490 9944 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalance_test_table_18" } modify_external_catalogs: true
I20240503 02:12:55.094069 13761 raft_consensus.cc:348] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.094517 13761 raft_consensus.cc:374] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.096386 13761 raft_consensus.cc:729] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.096861 13761 consensus_queue.cc:260] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.098717 13761 ts_tablet_manager.cc:1398] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:12:55.098495 9949 catalog_manager.cc:5484] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:55.099381 13761 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:55.102872 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 85f7e94021d446589390c7fa7f2940d3
I20240503 02:12:55.103608 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet d946fe77359940c1a5c9d4f2fff6e123
I20240503 02:12:55.104103 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 02b547bf800642deb59b168b1497f256
I20240503 02:12:55.104342 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 85f7e94021d446589390c7fa7f2940d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:55.104795 13619 tablet_replica.cc:330] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:55.105072 13619 raft_consensus.cc:2229] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.105285 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet d946fe77359940c1a5c9d4f2fff6e123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:55.105453 13619 raft_consensus.cc:2258] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.106076 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 85f7e94021d446589390c7fa7f2940d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:55.106376 13619 ts_tablet_manager.cc:1875] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.106470 13397 tablet_replica.cc:330] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:55.106691 13397 raft_consensus.cc:2229] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.107012 13397 raft_consensus.cc:2258] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.107837 13397 ts_tablet_manager.cc:1875] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.111757 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet d946fe77359940c1a5c9d4f2fff6e123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:55.112124 13665 tablet_replica.cc:330] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:55.112205 10073 tablet_service.cc:1514] Processing DeleteTablet for tablet 85f7e94021d446589390c7fa7f2940d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:55.112457 13665 raft_consensus.cc:2229] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.112614 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b547bf800642deb59b168b1497f256 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:55.112748 13665 raft_consensus.cc:2258] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.113457 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b547bf800642deb59b168b1497f256 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:12:55.113648 13665 ts_tablet_manager.cc:1875] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.105072 10471 tablet_service.cc:1514] Processing DeleteTablet for tablet 02b547bf800642deb59b168b1497f256 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:55.116662 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet d946fe77359940c1a5c9d4f2fff6e123 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:55.117012 13628 tablet_replica.cc:330] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:55.117259 13628 raft_consensus.cc:2229] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:55.117700 13628 raft_consensus.cc:2258] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.118736 13628 ts_tablet_manager.cc:1875] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.118921 9944 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table_new_name [id=b57f18a50b8548f79c96cda6440da2b8]). Expected version 3 got 2
I20240503 02:12:55.121151 13761 tablet_bootstrap.cc:654] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.126123 13397 ts_tablet_manager.cc:1888] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.126370 13397 log.cc:1196] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/85f7e94021d446589390c7fa7f2940d3
W20240503 02:12:55.126652 10424 consensus_peers.cc:485] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fb944e086c1f4287a1d4b5ec7aa87988. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:55.127138 13397 ts_tablet_manager.cc:1909] T 85f7e94021d446589390c7fa7f2940d3 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:55.128475 13174 raft_consensus.cc:1261] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.128989 13040 raft_consensus.cc:1261] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.129913 13781 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:55.130185 13397 tablet_replica.cc:330] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:12:55.130311 13786 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:55.130489 13397 raft_consensus.cc:2229] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.130662 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 85f7e94021d446589390c7fa7f2940d3 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.130777 13397 raft_consensus.cc:2258] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.131723 13397 ts_tablet_manager.cc:1875] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.132495 13823 tablet.cc:1722] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:55.133741 13761 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:55.134601 13761 ts_tablet_manager.cc:1367] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.002s
I20240503 02:12:55.135512 13761 raft_consensus.cc:348] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.135810 13761 raft_consensus.cc:374] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.138280 13761 raft_consensus.cc:729] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.138789 13761 consensus_queue.cc:260] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.139752 13761 ts_tablet_manager.cc:1398] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:12:55.144258 13665 ts_tablet_manager.cc:1888] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.149340 13628 ts_tablet_manager.cc:1888] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.149652 13628 log.cc:1196] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/85f7e94021d446589390c7fa7f2940d3
I20240503 02:12:55.149164 13487 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:12:55.149850 13665 log.cc:1196] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/d946fe77359940c1a5c9d4f2fff6e123
I20240503 02:12:55.150413 13665 ts_tablet_manager.cc:1909] T d946fe77359940c1a5c9d4f2fff6e123 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:12:55.150100 13040 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:12:55.151111 13628 ts_tablet_manager.cc:1909] T 85f7e94021d446589390c7fa7f2940d3 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:55.153658 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet d946fe77359940c1a5c9d4f2fff6e123 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.154053 13628 tablet_replica.cc:330] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:55.154057 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 85f7e94021d446589390c7fa7f2940d3 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.154340 13628 raft_consensus.cc:2229] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.154564 13397 ts_tablet_manager.cc:1888] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.154629 13628 raft_consensus.cc:2258] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.154774 13397 log.cc:1196] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/02b547bf800642deb59b168b1497f256
I20240503 02:12:55.155277 13397 ts_tablet_manager.cc:1909] T 02b547bf800642deb59b168b1497f256 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:12:55.155578 13628 ts_tablet_manager.cc:1875] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.156661 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 02b547bf800642deb59b168b1497f256 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.159529 9944 catalog_manager.cc:5484] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:55.161943 13619 ts_tablet_manager.cc:1888] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.162225 13619 log.cc:1196] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/85f7e94021d446589390c7fa7f2940d3
I20240503 02:12:55.162887 13619 ts_tablet_manager.cc:1909] T 85f7e94021d446589390c7fa7f2940d3 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:55.164225 13619 tablet_replica.cc:330] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:55.164564 13619 raft_consensus.cc:2229] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.164846 13619 raft_consensus.cc:2258] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.164912 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 85f7e94021d446589390c7fa7f2940d3 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.166129 13619 ts_tablet_manager.cc:1875] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.171757 13628 ts_tablet_manager.cc:1888] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.172264 13628 log.cc:1196] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/02b547bf800642deb59b168b1497f256
I20240503 02:12:55.173055 13628 ts_tablet_manager.cc:1909] T 02b547bf800642deb59b168b1497f256 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:55.174479 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 02b547bf800642deb59b168b1497f256 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.175014 13628 tablet_replica.cc:330] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:55.175328 13628 raft_consensus.cc:2229] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:55.175918 13628 raft_consensus.cc:2258] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.176842 13628 ts_tablet_manager.cc:1875] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.178647 13619 ts_tablet_manager.cc:1888] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.178859 13619 log.cc:1196] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/d946fe77359940c1a5c9d4f2fff6e123
I20240503 02:12:55.179385 13619 ts_tablet_manager.cc:1909] T d946fe77359940c1a5c9d4f2fff6e123 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:55.180729 13619 tablet_replica.cc:330] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:55.180658 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet d946fe77359940c1a5c9d4f2fff6e123 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.180999 13619 raft_consensus.cc:2229] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:55.181430 13619 raft_consensus.cc:2258] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.182312 13619 ts_tablet_manager.cc:1875] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.185812 13628 ts_tablet_manager.cc:1888] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.186061 13628 log.cc:1196] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/d946fe77359940c1a5c9d4f2fff6e123
I20240503 02:12:55.186582 13628 ts_tablet_manager.cc:1909] T d946fe77359940c1a5c9d4f2fff6e123 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:55.187913 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet d946fe77359940c1a5c9d4f2fff6e123 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.189172 13619 ts_tablet_manager.cc:1888] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20240503 02:12:55.189378 13619 log.cc:1196] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/02b547bf800642deb59b168b1497f256
I20240503 02:12:55.190093 13619 ts_tablet_manager.cc:1909] T 02b547bf800642deb59b168b1497f256 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:55.189894 9946 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:55.191478 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 02b547bf800642deb59b168b1497f256 (table rebalance_test_table_18 [id=96fac9ff2724483d85bc770dda5d97a6]) successfully deleted
I20240503 02:12:55.190500 9946 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:55.192515 13174 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:12:55.197414 9946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 7f054f44932640bdab328226673614ac
I20240503 02:12:55.198530 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f054f44932640bdab328226673614ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:12:55.198872 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f054f44932640bdab328226673614ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:12:55.199633 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet 7f054f44932640bdab328226673614ac with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:55 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:12:55.220832 13835 tablet_replica.cc:330] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:12:55.221496 13835 raft_consensus.cc:2229] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.223476 13834 tablet_replica.cc:330] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:12:55.223745 13836 tablet_replica.cc:330] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:12:55.223925 13834 raft_consensus.cc:2229] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:55.223996 13836 raft_consensus.cc:2229] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:55.224340 13836 raft_consensus.cc:2258] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.224566 13834 raft_consensus.cc:2258] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.225224 13835 raft_consensus.cc:2258] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:55.226029 13836 ts_tablet_manager.cc:1875] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.226449 13834 ts_tablet_manager.cc:1875] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.232801 13835 ts_tablet_manager.cc:1875] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:55.236320 13836 ts_tablet_manager.cc:1888] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:55.236361 13834 ts_tablet_manager.cc:1888] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:55.236588 13834 log.cc:1196] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/7f054f44932640bdab328226673614ac
I20240503 02:12:55.236779 13836 log.cc:1196] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/7f054f44932640bdab328226673614ac
I20240503 02:12:55.237322 13834 ts_tablet_manager.cc:1909] T 7f054f44932640bdab328226673614ac P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:12:55.238318 13836 ts_tablet_manager.cc:1909] T 7f054f44932640bdab328226673614ac P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:12:55.238948 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table_new_name [id=b57f18a50b8548f79c96cda6440da2b8]) successfully deleted
I20240503 02:12:55.239796 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table_new_name [id=b57f18a50b8548f79c96cda6440da2b8]) successfully deleted
I20240503 02:12:55.241683 13835 ts_tablet_manager.cc:1888] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:55.241837 13835 log.cc:1196] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/7f054f44932640bdab328226673614ac
I20240503 02:12:55.242286 13835 ts_tablet_manager.cc:1909] T 7f054f44932640bdab328226673614ac P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:12:55.243394 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 7f054f44932640bdab328226673614ac (table rebalancer_dynamic_table_new_name [id=b57f18a50b8548f79c96cda6440da2b8]) successfully deleted
I20240503 02:12:55.243319 10094 consensus_queue.cc:237] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:12:55.246894 10357 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:55.247149 10490 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:55.247918 13629 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:55.248337 13636 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:55.251996 13629 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:12:55.254128 10357 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
W20240503 02:12:55.256940 10026 consensus_peers.cc:485] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36457442a8e24ece9a1c2d8cdab89b78. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:55.256491 10490 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:12:55.260182 9946 catalog_manager.cc:5484] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:55.283756 12908 consensus_queue.cc:237] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:12:55.286926 13174 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.288076 13712 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:55.289712 10490 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.290585 13712 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:55.303869 13173 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:12:55.303102 13805 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:12:55.304520 10490 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:12:55.308027 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:55.310910 9944 catalog_manager.cc:5484] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
W20240503 02:12:55.326656 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67be061441154fbba1e227d32474cd5e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:55.331902 13853 ts_tablet_manager.cc:902] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:12:55.341270 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 4d823b46df0e48daa1f865bdcdb73798 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981]), partition=
I20240503 02:12:55.341367 13805 raft_consensus.cc:482] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:55.341558 13805 raft_consensus.cc:504] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.341845 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d823b46df0e48daa1f865bdcdb73798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.342383 13805 leader_election.cc:288] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:55.342900 12888 tablet_service.cc:1467] Processing CreateTablet for tablet 4d823b46df0e48daa1f865bdcdb73798 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981]), partition=
I20240503 02:12:55.343348 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d823b46df0e48daa1f865bdcdb73798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.345003 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 4d823b46df0e48daa1f865bdcdb73798 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981]), partition=
I20240503 02:12:55.345438 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d823b46df0e48daa1f865bdcdb73798. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.347847 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd081871c430470b95fce8f19634dc2e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:55.348186 13310 raft_consensus.cc:2379] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in current term 1: Already voted for candidate c6ec2f1283994a3089741715f811c850 in this term.
I20240503 02:12:55.348270 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd081871c430470b95fce8f19634dc2e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:55.349999 12840 leader_election.cc:302] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850
I20240503 02:12:55.350642 13805 raft_consensus.cc:3037] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.353482 13805 raft_consensus.cc:2726] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:55.354315 10094 consensus_queue.cc:237] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:12:55.357913 13761 tablet_bootstrap.cc:492] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:55.358039 10489 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:55.359723 13636 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:55.360853 13761 tablet_bootstrap.cc:654] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
W20240503 02:12:55.362519 10029 consensus_peers.cc:485] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc0a049eeb0047558d3c9eb370467c53. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:55.364228 10357 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:55.366755 13629 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:55.368995 13629 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:12:55.369889 13864 tablet_bootstrap.cc:492] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:55.372996 13861 tablet_bootstrap.cc:492] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:55.372576 10357 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:12:55.373781 13761 tablet_bootstrap.cc:492] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:55.374043 13761 ts_tablet_manager.cc:1367] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.001s
I20240503 02:12:55.376637 13761 raft_consensus.cc:348] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.377259 13761 raft_consensus.cc:374] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.377450 13761 raft_consensus.cc:729] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.377909 13761 consensus_queue.cc:260] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.378180 13864 tablet_bootstrap.cc:654] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.380512 13761 ts_tablet_manager.cc:1398] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.006s user 0.002s sys 0.002s
I20240503 02:12:55.380816 9944 catalog_manager.cc:5484] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:12:55.382408 10490 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:12:55.390352 13861 tablet_bootstrap.cc:654] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.396800 13864 tablet_bootstrap.cc:492] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:55.398844 13864 ts_tablet_manager.cc:1367] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20240503 02:12:55.401234 13864 raft_consensus.cc:348] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.401662 13864 raft_consensus.cc:374] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.402060 13864 raft_consensus.cc:729] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.402299 13864 consensus_queue.cc:260] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.403277 13864 ts_tablet_manager.cc:1398] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:55.400414 13861 tablet_bootstrap.cc:492] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:55.411994 13861 ts_tablet_manager.cc:1367] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.001s
I20240503 02:12:55.413509 13861 raft_consensus.cc:348] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.413977 13861 raft_consensus.cc:374] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.414137 13861 raft_consensus.cc:729] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.414539 13861 consensus_queue.cc:260] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.415514 13861 ts_tablet_manager.cc:1398] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:12:55.427070 13778 raft_consensus.cc:482] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:55.427470 13778 raft_consensus.cc:504] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.428524 13778 leader_election.cc:288] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:55.429436 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d823b46df0e48daa1f865bdcdb73798" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:55.429845 12908 raft_consensus.cc:2454] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:55.432080 13243 leader_election.cc:302] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:55.433676 13778 raft_consensus.cc:2781] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:55.433863 13778 raft_consensus.cc:482] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:55.434037 13778 raft_consensus.cc:3037] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.435829 13853 tablet_copy_client.cc:308] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:12:55.437961 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:45605)
I20240503 02:12:55.438256 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 1fa4eb577d7f42fd8e4a9a4391a0546f from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:45605: session id = 8d454a3607d042dc9559ed59c3ccd84b-1fa4eb577d7f42fd8e4a9a4391a0546f
I20240503 02:12:55.442229 10380 tablet_copy_source_session.cc:205] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:55.443787 13778 raft_consensus.cc:504] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.446379 13778 leader_election.cc:288] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:55.446645 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d823b46df0e48daa1f865bdcdb73798" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:55.446985 12908 raft_consensus.cc:3037] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.446578 13853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.451097 12908 raft_consensus.cc:2454] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:55.452669 13243 leader_election.cc:302] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:55.453064 13778 raft_consensus.cc:2781] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:55.453363 13778 raft_consensus.cc:686] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:12:55.454409 13778 consensus_queue.cc:237] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:12:55.464650 13853 tablet_copy_client.cc:783] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:55.464888 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d823b46df0e48daa1f865bdcdb73798" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:55.465214 13853 tablet_copy_client.cc:647] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:55.465198 10094 raft_consensus.cc:2454] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:12:55.465098 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d823b46df0e48daa1f865bdcdb73798" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:55.465405 10093 raft_consensus.cc:3037] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.468528 13853 tablet_copy_client.cc:523] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:55.473085 10093 raft_consensus.cc:2454] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:12:55.475033 13853 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:55.478127 9949 catalog_manager.cc:5484] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:55.498384 10490 consensus_queue.cc:237] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:12:55.502375 13310 raft_consensus.cc:1261] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.503232 13487 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:55.503584 10225 raft_consensus.cc:1261] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:12:55.505790 10427 consensus_peers.cc:485] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1578111fde6e43fd853142a48eb34ef5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:55.506661 13781 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:55.532372 13487 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:12:55.533432 13310 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:12:55.540964 9949 catalog_manager.cc:5484] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:12:55.543591 10226 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:12:55.548975 13781 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.549628 13639 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.560891 13787 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.562816 13853 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:55.570511 13853 tablet_bootstrap.cc:492] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:12:55.571868 13853 ts_tablet_manager.cc:1367] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.097s user 0.058s sys 0.012s
I20240503 02:12:55.573374 13853 raft_consensus.cc:348] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:12:55.575390 13853 raft_consensus.cc:729] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:12:55.575910 13853 consensus_queue.cc:260] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:12:55.586249 13853 ts_tablet_manager.cc:1398] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240503 02:12:55.588160 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-1fa4eb577d7f42fd8e4a9a4391a0546f received from {username='slave'} at 127.4.5.70:45605
I20240503 02:12:55.588371 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-1fa4eb577d7f42fd8e4a9a4391a0546f on tablet 1fa4eb577d7f42fd8e4a9a4391a0546f with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:12:55.595584 13890 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.612397 13487 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.619027 13805 raft_consensus.cc:482] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:55.619275 13805 raft_consensus.cc:504] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.620059 13805 leader_election.cc:288] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:55.620536 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:55.621640 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a142900b5c4909ad16b4abc70ee952" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:55.621886 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09a142900b5c4909ad16b4abc70ee952" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:55.622799 12842 leader_election.cc:302] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 8d454a3607d042dc9559ed59c3ccd84b, eff8e8d11a4447218741bb994aa41082
I20240503 02:12:55.623198 13805 raft_consensus.cc:2726] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:55.634541 12906 raft_consensus.cc:3037] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.639583 9947 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:55.650590 10093 raft_consensus.cc:1261] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.651401 12906 raft_consensus.cc:1261] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:55.651770 13795 consensus_queue.cc:1035] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.652057 13778 consensus_queue.cc:1035] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:55.665351 13290 tablet_service.cc:1467] Processing CreateTablet for tablet eb84e273efb54949811ab65950e93192 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.665331 10207 tablet_service.cc:1467] Processing CreateTablet for tablet eb84e273efb54949811ab65950e93192 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.665890 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb84e273efb54949811ab65950e93192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.665917 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb84e273efb54949811ab65950e93192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.640070 13787 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.028s
I20240503 02:12:55.670663 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2b9fd313794bbf85dfc1b423539e57 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.671310 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2b9fd313794bbf85dfc1b423539e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.671466 10074 tablet_service.cc:1467] Processing CreateTablet for tablet f1f1661a75a74746a4485b88b8e5b185 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.672047 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f1661a75a74746a4485b88b8e5b185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.676904 10471 tablet_service.cc:1467] Processing CreateTablet for tablet f1f1661a75a74746a4485b88b8e5b185 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.677608 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f1661a75a74746a4485b88b8e5b185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.679989 12888 tablet_service.cc:1467] Processing CreateTablet for tablet f1f1661a75a74746a4485b88b8e5b185 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.681622 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f1661a75a74746a4485b88b8e5b185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.714365 9947 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 4d823b46df0e48daa1f865bdcdb73798 (table rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981]). Expected version 1 got 0
I20240503 02:12:55.719745 13861 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:55.721974 13861 tablet_bootstrap.cc:654] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.723112 9946 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet 4d823b46df0e48daa1f865bdcdb73798 (table rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981]). Expected version 1 got 0
I20240503 02:12:55.724807 13040 raft_consensus.cc:1203] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:12:55.730638 13904 mvcc.cc:205] Tried to move back new op lower bound from 7023420930655145984 to 7023420929871409152. Current Snapshot: MvccSnapshot[applied={T|T < 7023420930655145984}]
I20240503 02:12:55.742270 13864 tablet_bootstrap.cc:492] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:55.744920 13761 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:55.745332 13864 tablet_bootstrap.cc:654] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.747249 13761 tablet_bootstrap.cc:654] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.750299 13913 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:55.752905 13909 tablet.cc:1722] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:55.754081 13909 tablet.cc:1620] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:55.755419 9949 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 4d823b46df0e48daa1f865bdcdb73798 (table rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981]). Expected version 1 got 0
I20240503 02:12:55.758096 13908 tablet.cc:1722] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:55.758847 13908 tablet.cc:1620] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:55.759447 13912 tablet.cc:1722] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:55.759917 13907 tablet_bootstrap.cc:492] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:55.760948 13912 tablet.cc:1620] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:12:55.763935 13861 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:55.764142 13861 ts_tablet_manager.cc:1367] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.006s
I20240503 02:12:55.764189 9947 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet 4d823b46df0e48daa1f865bdcdb73798 (table rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981]). Expected version 1 got 0
I20240503 02:12:55.765221 13861 raft_consensus.cc:348] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.765609 13861 raft_consensus.cc:374] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.765714 13861 raft_consensus.cc:729] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.766010 13861 consensus_queue.cc:260] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.766616 13861 ts_tablet_manager.cc:1398] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:12:55.766781 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981] alter complete (version 1)
I20240503 02:12:55.766959 13861 tablet_bootstrap.cc:492] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:55.770711 13913 tablet_bootstrap.cc:654] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.770923 13905 ts_tablet_manager.cc:902] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:55.775833 13019 tablet_service.cc:1467] Processing CreateTablet for tablet eb84e273efb54949811ab65950e93192 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.776398 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb84e273efb54949811ab65950e93192. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.777045 13907 tablet_bootstrap.cc:654] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.778031 13761 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:55.778241 13761 ts_tablet_manager.cc:1367] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.006s
I20240503 02:12:55.779402 13761 raft_consensus.cc:348] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.779656 13761 raft_consensus.cc:374] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.779731 13761 raft_consensus.cc:729] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.780054 13761 consensus_queue.cc:260] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.780910 13761 ts_tablet_manager.cc:1398] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:55.782536 13861 tablet_bootstrap.cc:654] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.800004 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2b9fd313794bbf85dfc1b423539e57 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.800729 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2b9fd313794bbf85dfc1b423539e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.805181 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2b9fd313794bbf85dfc1b423539e57 (DEFAULT_TABLE table=rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:55.805884 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2b9fd313794bbf85dfc1b423539e57. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.819339 13922 tablet_bootstrap.cc:492] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:55.823770 13907 tablet_bootstrap.cc:492] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:55.824105 13907 ts_tablet_manager.cc:1367] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.064s user 0.005s sys 0.005s
I20240503 02:12:55.826917 13907 raft_consensus.cc:348] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.827347 13907 raft_consensus.cc:374] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.827508 13907 raft_consensus.cc:729] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.829608 13907 consensus_queue.cc:260] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.832630 13907 ts_tablet_manager.cc:1398] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:12:55.838668 13922 tablet_bootstrap.cc:654] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.842375 13921 ts_tablet_manager.cc:902] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:55.844849 13861 tablet_bootstrap.cc:492] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:55.845222 13861 ts_tablet_manager.cc:1367] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.005s
I20240503 02:12:55.846494 13861 raft_consensus.cc:348] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.846947 13861 raft_consensus.cc:374] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.847110 13861 raft_consensus.cc:729] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.847574 13861 consensus_queue.cc:260] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.850149 13861 ts_tablet_manager.cc:1398] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:12:55.861371 13864 tablet_bootstrap.cc:492] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:55.861743 13864 ts_tablet_manager.cc:1367] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.120s user 0.003s sys 0.008s
I20240503 02:12:55.848205 13922 tablet_bootstrap.cc:492] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:55.863181 13922 ts_tablet_manager.cc:1367] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.002s
I20240503 02:12:55.865820 13922 raft_consensus.cc:348] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.867041 13864 raft_consensus.cc:348] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.867430 13922 raft_consensus.cc:374] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.867436 13864 raft_consensus.cc:374] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.867555 13864 raft_consensus.cc:729] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.867760 13922 raft_consensus.cc:729] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.867920 13864 consensus_queue.cc:260] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.868611 13864 ts_tablet_manager.cc:1398] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:55.876500 13920 tablet_bootstrap.cc:492] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:55.868548 13922 consensus_queue.cc:260] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.878567 13922 ts_tablet_manager.cc:1398] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.015s user 0.003s sys 0.000s
I20240503 02:12:55.879236 13913 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:55.879494 13913 ts_tablet_manager.cc:1367] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.130s user 0.002s sys 0.003s
I20240503 02:12:55.880909 13913 raft_consensus.cc:348] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.881227 13913 raft_consensus.cc:374] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.881387 13913 raft_consensus.cc:729] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.881685 13913 consensus_queue.cc:260] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.884889 13920 tablet_bootstrap.cc:654] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.890476 13913 ts_tablet_manager.cc:1398] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20240503 02:12:55.892582 13905 tablet_copy_client.cc:308] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:12:55.898819 9947 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:55.901180 13921 tablet_copy_client.cc:308] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:12:55.902956 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet fb944e086c1f4287a1d4b5ec7aa87988 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:41551)
I20240503 02:12:55.903242 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet fb944e086c1f4287a1d4b5ec7aa87988 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:41551: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-fb944e086c1f4287a1d4b5ec7aa87988
I20240503 02:12:55.910933 10511 tablet_copy_source_session.cc:205] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:55.914755 13928 ts_tablet_manager.cc:902] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:55.915843 13905 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb944e086c1f4287a1d4b5ec7aa87988. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.917989 13853 ts_tablet_manager.cc:902] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:55.924837 13912 tablet.cc:1722] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:55.925460 13912 tablet.cc:1620] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:12:55.929791 13920 tablet_bootstrap.cc:492] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:55.930210 13920 ts_tablet_manager.cc:1367] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.003s
I20240503 02:12:55.931360 13920 raft_consensus.cc:348] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.931694 13920 raft_consensus.cc:374] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:55.931836 13920 raft_consensus.cc:729] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:55.932194 13920 consensus_queue.cc:260] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:55.934154 13908 tablet.cc:1722] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:55.934813 13908 tablet.cc:1620] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:55.936156 13920 ts_tablet_manager.cc:1398] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:55.936645 13909 tablet.cc:1722] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:55.937264 13909 tablet.cc:1620] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:55.940693 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 36457442a8e24ece9a1c2d8cdab89b78 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:56811)
I20240503 02:12:55.940980 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 36457442a8e24ece9a1c2d8cdab89b78 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:56811: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-36457442a8e24ece9a1c2d8cdab89b78
I20240503 02:12:55.944949 10115 tablet_copy_source_session.cc:205] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:55.946720 13920 tablet_bootstrap.cc:492] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:55.949050 13920 tablet_bootstrap.cc:654] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:55.950927 13921 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36457442a8e24ece9a1c2d8cdab89b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:55.957036 9947 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=f96882db885742fe970b4f1f9163a981] alter complete (version 2)
I20240503 02:12:55.964620 13805 raft_consensus.cc:482] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:55.964866 13805 raft_consensus.cc:504] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.965760 13805 leader_election.cc:288] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:55.965881 13921 tablet_copy_client.cc:783] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:55.966245 13921 tablet_copy_client.cc:647] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:55.972546 13905 tablet_copy_client.cc:783] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:55.973383 13905 tablet_copy_client.cc:647] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:55.973621 13921 tablet_copy_client.cc:523] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:55.974284 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f1661a75a74746a4485b88b8e5b185" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:12:55.974567 10490 raft_consensus.cc:2454] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:55.975939 12840 leader_election.cc:302] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:55.976362 13805 raft_consensus.cc:2781] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:55.976532 13805 raft_consensus.cc:482] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:55.976635 13805 raft_consensus.cc:3037] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.979568 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f1661a75a74746a4485b88b8e5b185" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:55.979849 10093 raft_consensus.cc:2454] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:55.980408 13805 raft_consensus.cc:504] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:55.981364 13805 leader_election.cc:288] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:55.982097 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f1661a75a74746a4485b88b8e5b185" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:55.982332 10490 raft_consensus.cc:3037] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.983211 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f1f1661a75a74746a4485b88b8e5b185" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:55.983422 10093 raft_consensus.cc:3037] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:55.992132 10490 raft_consensus.cc:2454] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:55.994002 10093 raft_consensus.cc:2454] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:55.994714 12840 leader_election.cc:302] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:55.995131 13805 raft_consensus.cc:2781] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:55.995867 13921 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:55.998937 13905 tablet_copy_client.cc:523] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:55.998684 13928 tablet_copy_client.cc:308] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:12:56.001199 13920 tablet_bootstrap.cc:492] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:56.001432 13920 ts_tablet_manager.cc:1367] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.000s
I20240503 02:12:56.001729 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet 67be061441154fbba1e227d32474cd5e from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:52841)
I20240503 02:12:56.002003 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet 67be061441154fbba1e227d32474cd5e from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:52841: session id = eff8e8d11a4447218741bb994aa41082-67be061441154fbba1e227d32474cd5e
I20240503 02:12:56.002390 13920 raft_consensus.cc:348] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:56.002622 13920 raft_consensus.cc:374] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.002708 13920 raft_consensus.cc:729] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.003028 13920 consensus_queue.cc:260] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:56.003645 13920 ts_tablet_manager.cc:1398] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:56.004109 13905 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:56.006474 13805 raft_consensus.cc:686] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:12:56.007088 13805 consensus_queue.cc:237] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.008394 12928 tablet_copy_source_session.cc:205] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:56.013530 13928 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67be061441154fbba1e227d32474cd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.031021 13928 tablet_copy_client.cc:783] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:56.031553 13928 tablet_copy_client.cc:647] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:56.036338 13853 tablet_copy_client.cc:308] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:12:56.037487 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet fc0a049eeb0047558d3c9eb370467c53 from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:39295)
I20240503 02:12:56.037676 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet fc0a049eeb0047558d3c9eb370467c53 from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:39295: session id = 8d454a3607d042dc9559ed59c3ccd84b-fc0a049eeb0047558d3c9eb370467c53
I20240503 02:12:56.039242 13891 raft_consensus.cc:482] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.039449 13891 raft_consensus.cc:504] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:56.040432 13891 leader_election.cc:288] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:56.041343 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b9fd313794bbf85dfc1b423539e57" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:56.042449 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b9fd313794bbf85dfc1b423539e57" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:12:56.042737 10093 raft_consensus.cc:2454] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:56.042784 13173 raft_consensus.cc:2454] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:56.043403 12976 leader_election.cc:302] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:56.044191 13891 raft_consensus.cc:2781] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.044342 13891 raft_consensus.cc:482] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.044471 13891 raft_consensus.cc:3037] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.044736 10115 tablet_copy_source_session.cc:205] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:56.045890 13928 tablet_copy_client.cc:523] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:56.046308 13853 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0a049eeb0047558d3c9eb370467c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.049793 13891 raft_consensus.cc:504] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:56.051155 13928 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:56.051307 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b9fd313794bbf85dfc1b423539e57" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:56.051610 13173 raft_consensus.cc:3037] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.055320 13173 raft_consensus.cc:2454] T 2b2b9fd313794bbf85dfc1b423539e57 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:56.056174 12976 leader_election.cc:302] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:56.056540 13890 raft_consensus.cc:2781] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:56.056730 13890 raft_consensus.cc:686] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:56.057026 13890 consensus_queue.cc:237] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:56.057579 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2b9fd313794bbf85dfc1b423539e57" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:56.057898 10093 raft_consensus.cc:3037] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.061491 10093 raft_consensus.cc:2454] T 2b2b9fd313794bbf85dfc1b423539e57 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:56.062373 13891 leader_election.cc:288] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:12:56.069823 13905 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:56.073068 13905 tablet_bootstrap.cc:492] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:12:56.073698 13853 tablet_copy_client.cc:783] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:56.074193 13905 ts_tablet_manager.cc:1367] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.070s user 0.010s sys 0.003s
I20240503 02:12:56.074477 13853 tablet_copy_client.cc:647] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:56.075687 13905 raft_consensus.cc:348] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:12:56.077317 9949 catalog_manager.cc:5484] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:56.078639 13905 raft_consensus.cc:729] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:12:56.080662 9947 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:56.087810 13853 tablet_copy_client.cc:523] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:56.095954 13787 raft_consensus.cc:482] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.096149 13787 raft_consensus.cc:504] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:56.097196 13787 leader_election.cc:288] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:56.097810 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb84e273efb54949811ab65950e93192" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:56.098080 13040 raft_consensus.cc:2454] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:12:56.098321 13950 ts_tablet_manager.cc:902] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:12:56.101114 10162 leader_election.cc:302] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:56.102497 13912 tablet.cc:1722] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:56.103129 9949 catalog_manager.cc:5484] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:56.105206 13787 raft_consensus.cc:2781] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.105353 13787 raft_consensus.cc:482] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.105448 13787 raft_consensus.cc:3037] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.111735 13853 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:56.114567 13787 raft_consensus.cc:504] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:56.116156 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb84e273efb54949811ab65950e93192" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:56.116387 13040 raft_consensus.cc:3037] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.119335 13040 raft_consensus.cc:2454] T eb84e273efb54949811ab65950e93192 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:56.120286 10162 leader_election.cc:302] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:12:56.120687 13375 raft_consensus.cc:2781] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:56.124756 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=f96882db885742fe970b4f1f9163a981] alter complete (version 3)
I20240503 02:12:56.130044 13375 raft_consensus.cc:686] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:12:56.130398 13375 consensus_queue.cc:237] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:56.137094 13787 leader_election.cc:288] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:56.100728 13908 tablet.cc:1722] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:56.146715 13905 consensus_queue.cc:260] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:12:56.148912 13806 raft_consensus.cc:1050] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:12:56.151489 13806 consensus_queue.cc:237] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:12:56.161252 13905 ts_tablet_manager.cc:1398] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.087s user 0.000s sys 0.002s
I20240503 02:12:56.169131 9949 catalog_manager.cc:5484] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:56.178076 13040 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:56.179234 12719 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:56.189713 10490 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:56.191319 13228 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:56.210893 10093 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:56.211815 13228 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:56.235584 13921 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:56.235447 9946 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:56.235786 9946 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:56.236258 13921 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:12:56.236714 13921 ts_tablet_manager.cc:1367] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.257s user 0.076s sys 0.011s
I20240503 02:12:56.237954 13921 raft_consensus.cc:348] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:12:56.238284 13921 raft_consensus.cc:729] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:12:56.238569 13921 consensus_queue.cc:260] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:12:56.239480 13921 ts_tablet_manager.cc:1398] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:56.240824 9946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 4d823b46df0e48daa1f865bdcdb73798
I20240503 02:12:56.240801 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-36457442a8e24ece9a1c2d8cdab89b78 received from {username='slave'} at 127.4.5.69:56811
I20240503 02:12:56.240985 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-36457442a8e24ece9a1c2d8cdab89b78 on tablet 36457442a8e24ece9a1c2d8cdab89b78 with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:56.241818 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d823b46df0e48daa1f865bdcdb73798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:12:56.245486 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d823b46df0e48daa1f865bdcdb73798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:12:56.246474 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d823b46df0e48daa1f865bdcdb73798 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:56 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:56.248404 13979 tablet_replica.cc:330] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:12:56.248716 13979 raft_consensus.cc:2229] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:56.249364 13979 raft_consensus.cc:2258] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.250619 13979 ts_tablet_manager.cc:1875] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.252151 13981 tablet_replica.cc:330] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:56.252460 13981 raft_consensus.cc:2229] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.252810 13981 raft_consensus.cc:2258] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.257710 13981 ts_tablet_manager.cc:1875] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.258787 13228 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:56.260083 10490 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:56.267418 13980 tablet_replica.cc:330] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:12:56.268065 13950 tablet_copy_client.cc:308] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:12:56.269153 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-fb944e086c1f4287a1d4b5ec7aa87988 received from {username='slave'} at 127.4.5.72:41551
I20240503 02:12:56.269361 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-fb944e086c1f4287a1d4b5ec7aa87988 on tablet fb944e086c1f4287a1d4b5ec7aa87988 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:12:56.270641 13980 raft_consensus.cc:2229] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:56.271204 13980 pending_rounds.cc:62] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Trying to abort 1 pending ops.
I20240503 02:12:56.271641 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet 1578111fde6e43fd853142a48eb34ef5 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:33045)
I20240503 02:12:56.271834 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet 1578111fde6e43fd853142a48eb34ef5 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:33045: session id = f93b103af083471b8e8c0376e9148aad-1578111fde6e43fd853142a48eb34ef5
I20240503 02:12:56.275481 10511 tablet_copy_source_session.cc:205] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:12:56.271412 13980 pending_rounds.cc:68] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Aborting op as it isn't in flight: id { term: 1 index: 4 } timestamp: 7023420932464320512 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "4d823b46df0e48daa1f865bdcdb73798" schema { columns { id: 10 name: "key" type: INT64 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } columns { id: 12 name: "a" type: STRING is_key: false is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 3 new_table_name: "rebalancer_dynamic_table_new_name" dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" new_extra_config { } }
I20240503 02:12:56.279597 13980 raft_consensus.cc:2258] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:56.279696 10093 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:56.280187 13950 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1578111fde6e43fd853142a48eb34ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.281832 9947 catalog_manager.cc:5484] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:56.283643 13980 ts_tablet_manager.cc:1875] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:56.293157 13040 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:56.294080 13979 ts_tablet_manager.cc:1888] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:56.294330 13979 log.cc:1196] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/4d823b46df0e48daa1f865bdcdb73798
I20240503 02:12:56.295089 13979 ts_tablet_manager.cc:1909] T 4d823b46df0e48daa1f865bdcdb73798 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:12:56.301365 13981 ts_tablet_manager.cc:1888] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:56.301570 13981 log.cc:1196] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/4d823b46df0e48daa1f865bdcdb73798
I20240503 02:12:56.301955 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 4d823b46df0e48daa1f865bdcdb73798 (table rebalancer_dynamic_table_new_name [id=f96882db885742fe970b4f1f9163a981]) successfully deleted
I20240503 02:12:56.302981 13981 ts_tablet_manager.cc:1909] T 4d823b46df0e48daa1f865bdcdb73798 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:12:56.304311 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 4d823b46df0e48daa1f865bdcdb73798 (table rebalancer_dynamic_table_new_name [id=f96882db885742fe970b4f1f9163a981]) successfully deleted
I20240503 02:12:56.308100 13928 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:56.312713 13928 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:12:56.313086 13928 ts_tablet_manager.cc:1367] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.262s user 0.009s sys 0.002s
I20240503 02:12:56.315297 13950 tablet_copy_client.cc:783] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:12:56.316239 13980 ts_tablet_manager.cc:1888] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:56.316175 13309 raft_consensus.cc:1203] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:12:56.320976 13950 tablet_copy_client.cc:647] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:12:56.322160 13980 log.cc:1196] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/4d823b46df0e48daa1f865bdcdb73798
I20240503 02:12:56.323172 13980 ts_tablet_manager.cc:1909] T 4d823b46df0e48daa1f865bdcdb73798 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:12:56.324604 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 4d823b46df0e48daa1f865bdcdb73798 (table rebalancer_dynamic_table_new_name [id=f96882db885742fe970b4f1f9163a981]) successfully deleted
I20240503 02:12:56.326191 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb84e273efb54949811ab65950e93192" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:56.326452 13310 raft_consensus.cc:3037] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.326421 13928 raft_consensus.cc:348] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:12:56.326714 13928 raft_consensus.cc:729] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:12:56.327004 13928 consensus_queue.cc:260] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:12:56.328430 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb84e273efb54949811ab65950e93192" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:56.329754 13928 ts_tablet_manager.cc:1398] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.016s user 0.000s sys 0.002s
I20240503 02:12:56.330082 13950 tablet_copy_client.cc:523] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:12:56.330835 13310 raft_consensus.cc:2454] T eb84e273efb54949811ab65950e93192 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:12:56.340137 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-67be061441154fbba1e227d32474cd5e received from {username='slave'} at 127.4.5.66:52841
I20240503 02:12:56.340358 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session eff8e8d11a4447218741bb994aa41082-67be061441154fbba1e227d32474cd5e on tablet 67be061441154fbba1e227d32474cd5e with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:12:56.343560 13950 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:56.344398 12908 raft_consensus.cc:1203] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:12:56.348646 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:56.361907 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 5250d7aed30640e19005df4d1b9f2dff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]), partition=
I20240503 02:12:56.361907 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 5250d7aed30640e19005df4d1b9f2dff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]), partition=
I20240503 02:12:56.362411 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5250d7aed30640e19005df4d1b9f2dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.362411 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5250d7aed30640e19005df4d1b9f2dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.365931 13019 tablet_service.cc:1467] Processing CreateTablet for tablet 5250d7aed30640e19005df4d1b9f2dff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]), partition=
I20240503 02:12:56.366482 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5250d7aed30640e19005df4d1b9f2dff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.372397 13913 tablet_bootstrap.cc:492] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:56.374902 13913 tablet_bootstrap.cc:654] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.389060 13950 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:56.389823 13950 tablet_bootstrap.cc:492] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:12:56.390198 13950 ts_tablet_manager.cc:1367] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.047s user 0.011s sys 0.001s
I20240503 02:12:56.391340 13950 raft_consensus.cc:348] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:12:56.397783 13950 raft_consensus.cc:729] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:12:56.398192 13950 consensus_queue.cc:260] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:12:56.399019 13950 ts_tablet_manager.cc:1398] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:12:56.401487 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-1578111fde6e43fd853142a48eb34ef5 received from {username='slave'} at 127.4.5.71:33045
I20240503 02:12:56.401654 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session f93b103af083471b8e8c0376e9148aad-1578111fde6e43fd853142a48eb34ef5 on tablet 1578111fde6e43fd853142a48eb34ef5 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:12:56.423413 13920 tablet_bootstrap.cc:492] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:56.425886 13920 tablet_bootstrap.cc:654] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.426285 13853 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:12:56.427011 13853 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:12:56.427382 13853 ts_tablet_manager.cc:1367] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.316s user 0.077s sys 0.037s
I20240503 02:12:56.430243 13913 tablet_bootstrap.cc:492] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:56.430765 13913 ts_tablet_manager.cc:1367] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.009s
I20240503 02:12:56.431918 13913 raft_consensus.cc:348] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.432600 13913 raft_consensus.cc:374] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.432754 13913 raft_consensus.cc:729] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.433511 13913 consensus_queue.cc:260] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.434509 13913 ts_tablet_manager.cc:1398] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:56.436369 13920 tablet_bootstrap.cc:492] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:56.436666 13920 ts_tablet_manager.cc:1367] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.006s
I20240503 02:12:56.437506 13853 raft_consensus.cc:348] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:12:56.437860 13853 raft_consensus.cc:729] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:12:56.437764 13920 raft_consensus.cc:348] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.438246 13920 raft_consensus.cc:374] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.438429 13920 raft_consensus.cc:729] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.438791 13920 consensus_queue.cc:260] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.438153 13853 consensus_queue.cc:260] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:12:56.441459 13853 ts_tablet_manager.cc:1398] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.014s user 0.005s sys 0.000s
I20240503 02:12:56.441970 13920 ts_tablet_manager.cc:1398] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20240503 02:12:56.442907 13993 tablet_bootstrap.cc:492] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:56.443253 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-fc0a049eeb0047558d3c9eb370467c53 received from {username='slave'} at 127.4.5.70:39295
I20240503 02:12:56.443419 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-fc0a049eeb0047558d3c9eb370467c53 on tablet fc0a049eeb0047558d3c9eb370467c53 with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:12:56.448323 13993 tablet_bootstrap.cc:654] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.457028 13805 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.468681 13993 tablet_bootstrap.cc:492] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:56.468923 13993 ts_tablet_manager.cc:1367] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.007s
I20240503 02:12:56.469915 13993 raft_consensus.cc:348] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.470181 13993 raft_consensus.cc:374] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.470289 13993 raft_consensus.cc:729] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.470471 13174 raft_consensus.cc:1203] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:12:56.470571 13993 consensus_queue.cc:260] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.473920 13993 ts_tablet_manager.cc:1398] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:12:56.475255 13942 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.485656 10226 raft_consensus.cc:1203] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:12:56.496083 13890 consensus_queue.cc:1035] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.528612 13040 raft_consensus.cc:1203] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:12:56.542070 13375 consensus_queue.cc:1035] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.560635 13891 consensus_queue.cc:1035] T 2b2b9fd313794bbf85dfc1b423539e57 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.572979 13781 raft_consensus.cc:482] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.573241 13781 raft_consensus.cc:504] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.574003 13781 leader_election.cc:288] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:56.575194 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5250d7aed30640e19005df4d1b9f2dff" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:56.575474 13040 raft_consensus.cc:2454] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:56.576197 10427 leader_election.cc:302] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:56.576690 13781 raft_consensus.cc:2781] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.576810 13781 raft_consensus.cc:482] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.576925 13781 raft_consensus.cc:3037] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.576998 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5250d7aed30640e19005df4d1b9f2dff" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:56.577348 13310 raft_consensus.cc:2454] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:56.582571 13962 consensus_queue.cc:1035] T eb84e273efb54949811ab65950e93192 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.585633 13781 raft_consensus.cc:504] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.587143 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5250d7aed30640e19005df4d1b9f2dff" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:56.587378 13039 raft_consensus.cc:3037] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.588901 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5250d7aed30640e19005df4d1b9f2dff" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:56.589181 13310 raft_consensus.cc:3037] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.590652 13039 raft_consensus.cc:2454] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:56.591923 10427 leader_election.cc:302] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:56.592275 13487 raft_consensus.cc:2781] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:56.592401 13781 leader_election.cc:288] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:56.592484 13487 raft_consensus.cc:686] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:56.592794 13487 consensus_queue.cc:237] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:56.600752 13310 raft_consensus.cc:2454] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:56.612751 9949 catalog_manager.cc:5484] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:56.663038 10357 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f"
dest_uuid: "47f901e617024baaa1865d12261ef84f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:38776
I20240503 02:12:56.663368 10357 raft_consensus.cc:595] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:56.705212 12719 raft_consensus.cc:979] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: : Instructing follower 031a0afb01b6469299fa829fa3467ef7 to start an election
I20240503 02:12:56.705441 13228 raft_consensus.cc:1067] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Signalling peer 031a0afb01b6469299fa829fa3467ef7 to start an election
I20240503 02:12:56.706511 10094 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f"
dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
from {username='slave'} at 127.4.5.67:60077
I20240503 02:12:56.706892 10094 raft_consensus.cc:482] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:56.707046 10094 raft_consensus.cc:3037] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:56.709542 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:56.710621 10094 raft_consensus.cc:504] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:12:56.712378 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:56.712718 10357 raft_consensus.cc:3032] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:56.712821 10357 raft_consensus.cc:729] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:56.713213 10357 consensus_queue.cc:260] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:12:56.713665 10357 raft_consensus.cc:3037] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:56.714175 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:12:56.714464 10490 raft_consensus.cc:3037] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:56.715210 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1fa4eb577d7f42fd8e4a9a4391a0546f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:56.715478 13040 raft_consensus.cc:3037] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:56.717510 10490 raft_consensus.cc:2454] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 2.
I20240503 02:12:56.718448 10357 raft_consensus.cc:2454] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 2.
I20240503 02:12:56.718748 13040 raft_consensus.cc:2454] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 2.
I20240503 02:12:56.719633 10029 leader_election.cc:302] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f, 8d454a3607d042dc9559ed59c3ccd84b; no voters:
I20240503 02:12:56.719696 10094 leader_election.cc:288] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:56.720065 13812 raft_consensus.cc:2781] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:56.720244 13812 raft_consensus.cc:686] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 2 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:56.720695 13812 consensus_queue.cc:237] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:12:56.724763 9949 catalog_manager.cc:5484] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 2 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:56.726559 10207 tablet_service.cc:1467] Processing CreateTablet for tablet bf4870715de449cda20ad5f2e02397c9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.727136 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4870715de449cda20ad5f2e02397c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.728042 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 0892c659fe1a43b796a73ee8b1783173 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.728574 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0892c659fe1a43b796a73ee8b1783173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.733505 12888 tablet_service.cc:1467] Processing CreateTablet for tablet bf4870715de449cda20ad5f2e02397c9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.735116 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4870715de449cda20ad5f2e02397c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.736958 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 46e5690a42d44500b3e6ca24d771affc (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.737412 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e5690a42d44500b3e6ca24d771affc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.738940 14041 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:56.742408 14041 tablet_bootstrap.cc:654] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.742522 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 0892c659fe1a43b796a73ee8b1783173 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.742913 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 46e5690a42d44500b3e6ca24d771affc (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.743103 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0892c659fe1a43b796a73ee8b1783173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.743300 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e5690a42d44500b3e6ca24d771affc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.742524 13290 tablet_service.cc:1467] Processing CreateTablet for tablet bf4870715de449cda20ad5f2e02397c9 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.746492 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bf4870715de449cda20ad5f2e02397c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.754175 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:56.756764 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 0892c659fe1a43b796a73ee8b1783173 (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.757314 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0892c659fe1a43b796a73ee8b1783173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.762799 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 46e5690a42d44500b3e6ca24d771affc (DEFAULT_TABLE table=rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:56.763438 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e5690a42d44500b3e6ca24d771affc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:56.768852 13993 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:56.769244 13040 raft_consensus.cc:1261] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:56.769958 13310 raft_consensus.cc:1261] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:56.770012 13786 consensus_queue.cc:1035] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.770797 13993 tablet_bootstrap.cc:654] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.771644 13786 consensus_queue.cc:1035] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:56.793532 13993 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:56.793841 13993 ts_tablet_manager.cc:1367] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20240503 02:12:56.796131 14047 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:56.796978 14048 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:56.800720 14000 mvcc.cc:205] Tried to move back new op lower bound from 7023420935237779456 to 7023420934526799872. Current Snapshot: MvccSnapshot[applied={T|T < 7023420935237779456}]
I20240503 02:12:56.805971 14047 tablet_bootstrap.cc:654] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.807757 14048 tablet_bootstrap.cc:654] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.813684 13993 raft_consensus.cc:348] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.815145 13993 raft_consensus.cc:374] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.815593 13993 raft_consensus.cc:729] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.817221 9949 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]). Expected version 1 got 0
I20240503 02:12:56.818616 14048 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:56.818814 14048 ts_tablet_manager.cc:1367] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.003s
I20240503 02:12:56.819355 14041 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:56.819403 9946 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]). Expected version 1 got 0
I20240503 02:12:56.819540 14041 ts_tablet_manager.cc:1367] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.081s user 0.014s sys 0.009s
I20240503 02:12:56.820540 14041 raft_consensus.cc:348] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.820974 13993 consensus_queue.cc:260] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.821749 13487 raft_consensus.cc:1050] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:12:56.821981 14048 raft_consensus.cc:348] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.822258 14048 raft_consensus.cc:374] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.822386 14048 raft_consensus.cc:729] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.822598 14046 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:56.822615 14041 raft_consensus.cc:374] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.822716 14041 raft_consensus.cc:729] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.823249 13487 consensus_queue.cc:237] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:12:56.824007 14041 consensus_queue.cc:260] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.825590 14047 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:56.825811 14047 ts_tablet_manager.cc:1367] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.000s
I20240503 02:12:56.826838 14047 raft_consensus.cc:348] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.827085 14047 raft_consensus.cc:374] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.827203 14047 raft_consensus.cc:729] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.827517 14047 consensus_queue.cc:260] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.829396 13993 ts_tablet_manager.cc:1398] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.035s user 0.001s sys 0.002s
I20240503 02:12:56.830747 14047 ts_tablet_manager.cc:1398] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:12:56.831166 14048 consensus_queue.cc:260] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.831887 14047 tablet_bootstrap.cc:492] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:12:56.831929 13993 tablet_bootstrap.cc:492] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:56.833701 14048 ts_tablet_manager.cc:1398] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
I20240503 02:12:56.834090 14048 tablet_bootstrap.cc:492] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:56.836086 14048 tablet_bootstrap.cc:654] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.844297 14041 ts_tablet_manager.cc:1398] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20240503 02:12:56.845183 13384 raft_consensus.cc:482] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:56.845422 13384 raft_consensus.cc:504] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.845604 14046 tablet_bootstrap.cc:654] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.845722 14049 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:56.846184 14047 tablet_bootstrap.cc:654] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.846403 13384 leader_election.cc:288] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:56.847410 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0892c659fe1a43b796a73ee8b1783173" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:56.847863 13310 raft_consensus.cc:2454] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:12:56.849074 13107 leader_election.cc:302] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:56.849769 13384 raft_consensus.cc:2781] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:56.849871 13384 raft_consensus.cc:482] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:56.849993 13384 raft_consensus.cc:3037] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.857409 13384 raft_consensus.cc:504] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.858656 14046 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:56.858844 13384 leader_election.cc:288] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:56.859392 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0892c659fe1a43b796a73ee8b1783173" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:56.859827 13310 raft_consensus.cc:3037] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:56.860085 14046 ts_tablet_manager.cc:1367] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.003s
I20240503 02:12:56.860569 9947 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]). Expected version 1 got 0
I20240503 02:12:56.868743 14046 raft_consensus.cc:348] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.869725 13310 raft_consensus.cc:2454] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:12:56.869865 14046 raft_consensus.cc:374] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.870560 13993 tablet_bootstrap.cc:654] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:56.870298 14046 raft_consensus.cc:729] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.871116 14046 consensus_queue.cc:260] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.871413 13107 leader_election.cc:302] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:56.872116 9947 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33] alter complete (version 1)
I20240503 02:12:56.872859 13384 raft_consensus.cc:2781] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:56.877672 14049 tablet.cc:1620] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:12:56.879843 14048 tablet_bootstrap.cc:492] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:56.880205 14048 ts_tablet_manager.cc:1367] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.002s
I20240503 02:12:56.881335 14048 raft_consensus.cc:348] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.881682 14048 raft_consensus.cc:374] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.881817 14048 raft_consensus.cc:729] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.882226 14048 consensus_queue.cc:260] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.884042 14048 ts_tablet_manager.cc:1398] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:12:56.884732 14046 ts_tablet_manager.cc:1398] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240503 02:12:56.898803 14050 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:56.902597 14050 tablet.cc:1620] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:56.907228 13384 raft_consensus.cc:686] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:12:56.907637 13384 consensus_queue.cc:237] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:56.908365 14047 tablet_bootstrap.cc:492] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:12:56.908664 14047 ts_tablet_manager.cc:1367] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.023s
I20240503 02:12:56.909858 14047 raft_consensus.cc:348] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.910215 14047 raft_consensus.cc:374] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.910356 14047 raft_consensus.cc:729] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.910858 14047 consensus_queue.cc:260] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.912935 14047 ts_tablet_manager.cc:1398] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:12:56.914368 9949 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]). Expected version 1 got 0
I20240503 02:12:56.918828 13039 raft_consensus.cc:1261] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:56.919497 13781 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:56.920073 13174 raft_consensus.cc:1261] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:56.920939 13781 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:56.924284 13812 raft_consensus.cc:1050] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:12:56.925729 13812 consensus_queue.cc:237] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:12:56.931143 12908 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:56.932045 13629 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:56.934939 13309 raft_consensus.cc:1261] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:56.936262 13487 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:56.936817 10357 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:56.937515 13629 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:56.948581 10489 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:56.950459 13812 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 02:12:56.951231 13993 tablet_bootstrap.cc:492] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:56.952157 14054 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:56.956908 14054 tablet.cc:1620] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:56.961424 13993 ts_tablet_manager.cc:1367] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.130s user 0.006s sys 0.000s
I20240503 02:12:56.966557 9949 catalog_manager.cc:5484] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 1, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:56.972210 9947 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]). Expected version 1 got 0
I20240503 02:12:56.978989 13993 raft_consensus.cc:348] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.981113 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:56.982302 13993 raft_consensus.cc:374] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:56.982888 13993 raft_consensus.cc:729] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:56.983539 13993 consensus_queue.cc:260] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:56.985795 13781 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:56.990077 13812 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:12:56.993736 9947 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33]). Expected version 2 got 1
I20240503 02:12:56.995971 10357 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:12:57.002313 13310 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:57.003304 13039 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:57.007125 13787 raft_consensus.cc:482] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.006947 9947 catalog_manager.cc:5484] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:12:57.007325 13787 raft_consensus.cc:504] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:57.008244 13787 leader_election.cc:288] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:57.011281 12908 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:12:57.015940 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0892c659fe1a43b796a73ee8b1783173" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:57.016382 13308 raft_consensus.cc:2379] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eff8e8d11a4447218741bb994aa41082 in current term 1: Already voted for candidate f93b103af083471b8e8c0376e9148aad in this term.
I20240503 02:12:57.027542 9952 catalog_manager.cc:5484] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:57.028775 14049 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.029426 14049 tablet.cc:1620] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:12:57.036289 13629 raft_consensus.cc:1050] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:12:57.037573 13629 consensus_queue.cc:237] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:12:57.042680 10225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0892c659fe1a43b796a73ee8b1783173" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:57.044026 10225 raft_consensus.cc:3037] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.045526 13039 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:57.046175 13629 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:57.047338 10225 raft_consensus.cc:2454] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:12:57.047358 10357 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:57.048094 13629 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:57.049499 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0892c659fe1a43b796a73ee8b1783173" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:57.049904 10226 raft_consensus.cc:2362] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1. Re-sending same reply.
I20240503 02:12:57.053838 10489 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:12:57.055745 13990 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:12:57.060544 10489 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
W20240503 02:12:57.070462 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.70:43707, user_credentials={real_user=slave}} blocked reactor thread for 59132.4us
I20240503 02:12:57.072080 14054 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.072747 13786 raft_consensus.cc:1050] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:12:57.072842 14054 tablet.cc:1620] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:57.073769 13786 consensus_queue.cc:237] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:12:57.079636 10226 raft_consensus.cc:1261] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:57.079654 13174 raft_consensus.cc:1261] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:57.080592 13786 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:12:57.083943 10424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 61536.9us
I20240503 02:12:57.085265 13786 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:57.086884 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=c854d2aa767e478a9b45fe3e92e81b33] alter complete (version 2)
I20240503 02:12:57.088043 13173 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:57.101297 13943 raft_consensus.cc:1050] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:12:57.103041 13943 consensus_queue.cc:237] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:12:57.103924 13990 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.109526 10489 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.113312 10358 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.117525 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0892c659fe1a43b796a73ee8b1783173" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:57.118824 10162 leader_election.cc:302] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eff8e8d11a4447218741bb994aa41082; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad
I20240503 02:12:57.119431 13375 raft_consensus.cc:2726] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:57.121378 9946 catalog_manager.cc:5484] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:57.123431 13309 raft_consensus.cc:1261] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:57.124509 13990 raft_consensus.cc:482] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.124701 13990 raft_consensus.cc:504] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.126161 13990 leader_election.cc:288] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:57.126298 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e5690a42d44500b3e6ca24d771affc" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:57.126585 12908 raft_consensus.cc:2454] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:57.127169 10026 leader_election.cc:302] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:57.127650 13990 raft_consensus.cc:2781] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:57.127839 13990 raft_consensus.cc:482] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:57.128006 13990 raft_consensus.cc:3037] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.132002 10225 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:57.132956 13942 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:57.133441 13172 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:57.134232 13942 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:57.135020 10490 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:12:57.135808 13942 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:57.151387 13990 raft_consensus.cc:504] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.155790 13990 leader_election.cc:288] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:57.157035 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e5690a42d44500b3e6ca24d771affc" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:57.157389 12908 raft_consensus.cc:3037] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.164425 12908 raft_consensus.cc:2454] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:57.165555 10026 leader_election.cc:302] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:57.166736 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e5690a42d44500b3e6ca24d771affc" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:57.168164 13990 raft_consensus.cc:2781] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:12:57.168370 10026 leader_election.cc:341] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:57.168828 13990 raft_consensus.cc:686] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:57.170826 13990 consensus_queue.cc:237] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.174435 10357 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:57.175436 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "46e5690a42d44500b3e6ca24d771affc" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:57.175379 13629 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:57.179068 14078 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:12:57.179481 13993 ts_tablet_manager.cc:1398] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.218s user 0.002s sys 0.001s
I20240503 02:12:57.179935 13993 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
W20240503 02:12:57.184818 10026 leader_election.cc:341] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:57.186158 9946 catalog_manager.cc:5484] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:57.190465 9949 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:57.192799 13039 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.195803 14050 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:57.196519 14050 tablet.cc:1620] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:57.197492 10489 consensus_queue.cc:237] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:12:57.202133 13039 raft_consensus.cc:1261] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:57.202941 14075 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:57.206521 13943 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:12:57.207806 14049 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:57.215962 10226 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:12:57.216601 9952 catalog_manager.cc:5484] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:57.221385 13993 tablet_bootstrap.cc:654] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.221657 14054 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:57.223944 10490 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:57.228056 13172 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:12:57.229372 10489 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:12:57.235563 12908 consensus_queue.cc:237] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:12:57.236730 10094 consensus_queue.cc:237] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:12:57.240176 13040 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:12:57.241542 14078 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:57.243095 10489 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:12:57.244508 13812 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:57.245016 13629 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:57.244733 13487 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.246234 13812 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:57.249575 10226 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:57.250466 13943 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:57.251149 13172 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:57.251848 13812 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.252548 13943 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:57.253453 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=c854d2aa767e478a9b45fe3e92e81b33] alter complete (version 3)
I20240503 02:12:57.254137 13174 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.256978 13942 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:12:57.257694 10225 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.260071 10490 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.265853 13040 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.268049 13174 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:12:57.270625 13308 raft_consensus.cc:1261] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:57.271448 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67be061441154fbba1e227d32474cd5e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:57.272807 10226 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:12:57.272981 9952 catalog_manager.cc:5484] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:57.273411 9949 catalog_manager.cc:5484] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.274039 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc0a049eeb0047558d3c9eb370467c53 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:57.274078 9944 catalog_manager.cc:5484] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:12:57.276945 9946 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table_new_name [id=c854d2aa767e478a9b45fe3e92e81b33]). Expected version 3 got 2
I20240503 02:12:57.281641 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0a049eeb0047558d3c9eb370467c53 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:57.276546 14079 consensus_queue.cc:1035] T fb944e086c1f4287a1d4b5ec7aa87988 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:57.289956 13308 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:57.292642 14050 tablet.cc:1722] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:57.301333 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 67be061441154fbba1e227d32474cd5e with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:57.309458 13384 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.310688 13993 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:57.311002 13993 ts_tablet_manager.cc:1367] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.131s user 0.007s sys 0.004s
I20240503 02:12:57.315887 13993 raft_consensus.cc:348] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.316483 13993 raft_consensus.cc:374] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.316673 13993 raft_consensus.cc:729] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.317157 13993 consensus_queue.cc:260] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.318495 13993 ts_tablet_manager.cc:1398] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:12:57.321908 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fb944e086c1f4287a1d4b5ec7aa87988 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:57.324483 13040 raft_consensus.cc:2932] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:57.328083 14099 tablet_replica.cc:330] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:57.328397 14099 raft_consensus.cc:2229] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.327980 13307 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.328792 14099 raft_consensus.cc:2258] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.330530 14102 tablet_replica.cc:330] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:57.330849 14102 raft_consensus.cc:2229] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.331171 14102 raft_consensus.cc:2258] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.333588 9944 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table_new_name [id=c854d2aa767e478a9b45fe3e92e81b33]). Expected version 3 got 2
I20240503 02:12:57.340752 13040 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:57.341002 9943 catalog_manager.cc:5484] T fb944e086c1f4287a1d4b5ec7aa87988 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:12:57.349076 14099 ts_tablet_manager.cc:1875] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:57.349005 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet fb944e086c1f4287a1d4b5ec7aa87988 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:12:57.350494 13629 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:57.350680 14102 ts_tablet_manager.cc:1875] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:57.356637 14107 tablet_replica.cc:330] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:12:57.359375 10491 consensus_queue.cc:237] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:12:57.360396 13805 raft_consensus.cc:482] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.360654 13805 raft_consensus.cc:504] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.361594 13805 leader_election.cc:288] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:57.361912 10226 raft_consensus.cc:1261] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:57.362682 14078 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:57.362880 13174 raft_consensus.cc:1261] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:57.363711 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4870715de449cda20ad5f2e02397c9" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:57.364033 13310 raft_consensus.cc:2454] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:57.364578 14078 consensus_queue.cc:1035] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:57.364691 14107 raft_consensus.cc:2229] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.365309 14075 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.368098 14107 raft_consensus.cc:2258] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.368546 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4870715de449cda20ad5f2e02397c9" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:12:57.368837 10226 raft_consensus.cc:2454] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:57.369261 10227 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.369812 14107 ts_tablet_manager.cc:1875] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:57.370141 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1578111fde6e43fd853142a48eb34ef5 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:57.373790 12840 leader_election.cc:302] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:57.375336 13805 raft_consensus.cc:2781] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:57.375490 13805 raft_consensus.cc:482] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:57.375591 13805 raft_consensus.cc:3037] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.372807 13174 raft_consensus.cc:2932] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.373507 13384 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.003s
I20240503 02:12:57.381767 13805 raft_consensus.cc:504] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.384347 14102 ts_tablet_manager.cc:1888] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:12:57.387161 14099 ts_tablet_manager.cc:1888] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:12:57.387600 14099 log.cc:1196] T fc0a049eeb0047558d3c9eb370467c53 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/fc0a049eeb0047558d3c9eb370467c53
I20240503 02:12:57.389052 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet fc0a049eeb0047558d3c9eb370467c53 (table rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]) successfully deleted
I20240503 02:12:57.390064 14102 log.cc:1196] T 67be061441154fbba1e227d32474cd5e P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/67be061441154fbba1e227d32474cd5e
I20240503 02:12:57.389767 9951 catalog_manager.cc:5484] T 1578111fde6e43fd853142a48eb34ef5 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:12:57.392364 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 67be061441154fbba1e227d32474cd5e (table rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]) successfully deleted
I20240503 02:12:57.400061 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4870715de449cda20ad5f2e02397c9" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:57.400406 13310 raft_consensus.cc:3037] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.404007 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet 1578111fde6e43fd853142a48eb34ef5 with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:12:57.399307 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bf4870715de449cda20ad5f2e02397c9" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:12:57.404275 13310 raft_consensus.cc:2454] T bf4870715de449cda20ad5f2e02397c9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:57.404327 10227 raft_consensus.cc:3037] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.404981 12840 leader_election.cc:302] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:57.405387 13943 raft_consensus.cc:2781] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:57.405555 13943 raft_consensus.cc:686] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:12:57.405879 13943 consensus_queue.cc:237] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.408254 10227 raft_consensus.cc:2454] T bf4870715de449cda20ad5f2e02397c9 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:57.412103 14111 tablet_replica.cc:330] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:12:57.412686 14111 raft_consensus.cc:2229] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.417712 14111 raft_consensus.cc:2258] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.418406 14107 ts_tablet_manager.cc:1888] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:12:57.418675 14107 log.cc:1196] T fb944e086c1f4287a1d4b5ec7aa87988 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/fb944e086c1f4287a1d4b5ec7aa87988
I20240503 02:12:57.419751 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet fb944e086c1f4287a1d4b5ec7aa87988 (table rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]) successfully deleted
I20240503 02:12:57.420588 13805 leader_election.cc:288] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:57.423518 14111 ts_tablet_manager.cc:1875] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:57.438367 14111 ts_tablet_manager.cc:1888] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:12:57.439283 14111 log.cc:1196] T 1578111fde6e43fd853142a48eb34ef5 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/1578111fde6e43fd853142a48eb34ef5
I20240503 02:12:57.440364 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 1578111fde6e43fd853142a48eb34ef5 (table rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]) successfully deleted
I20240503 02:12:57.440306 9949 catalog_manager.cc:5484] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:57.441571 9944 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:57.441850 9944 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:57.452087 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 5250d7aed30640e19005df4d1b9f2dff
I20240503 02:12:57.453615 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet 5250d7aed30640e19005df4d1b9f2dff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:12:57.454032 14111 tablet_replica.cc:330] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:12:57.454313 14111 raft_consensus.cc:2229] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.454298 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet 5250d7aed30640e19005df4d1b9f2dff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:12:57.454614 14111 raft_consensus.cc:2258] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.455042 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5250d7aed30640e19005df4d1b9f2dff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:57 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:57.455889 14111 ts_tablet_manager.cc:1875] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.456099 14102 tablet_replica.cc:330] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:57.456454 14102 raft_consensus.cc:2229] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:57.456984 14102 raft_consensus.cc:2258] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.458536 14102 ts_tablet_manager.cc:1875] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.460394 14120 tablet_replica.cc:330] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:12:57.460660 14120 raft_consensus.cc:2229] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.460923 14120 raft_consensus.cc:2258] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.465624 14111 ts_tablet_manager.cc:1888] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:57.466331 14102 ts_tablet_manager.cc:1888] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:57.466512 14102 log.cc:1196] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/5250d7aed30640e19005df4d1b9f2dff
I20240503 02:12:57.467003 14120 ts_tablet_manager.cc:1875] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:57.467945 14102 ts_tablet_manager.cc:1909] T 5250d7aed30640e19005df4d1b9f2dff P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:57.470300 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table_new_name [id=c854d2aa767e478a9b45fe3e92e81b33]) successfully deleted
I20240503 02:12:57.472615 14111 log.cc:1196] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/5250d7aed30640e19005df4d1b9f2dff
I20240503 02:12:57.473122 14111 ts_tablet_manager.cc:1909] T 5250d7aed30640e19005df4d1b9f2dff P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:12:57.474644 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table_new_name [id=c854d2aa767e478a9b45fe3e92e81b33]) successfully deleted
I20240503 02:12:57.475639 14120 ts_tablet_manager.cc:1888] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:57.475852 14120 log.cc:1196] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/5250d7aed30640e19005df4d1b9f2dff
I20240503 02:12:57.476508 14120 ts_tablet_manager.cc:1909] T 5250d7aed30640e19005df4d1b9f2dff P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:12:57.477969 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 5250d7aed30640e19005df4d1b9f2dff (table rebalancer_dynamic_table_new_name [id=c854d2aa767e478a9b45fe3e92e81b33]) successfully deleted
I20240503 02:12:57.517293 10094 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78"
dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54442
I20240503 02:12:57.517657 10094 raft_consensus.cc:595] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:57.569559 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:57.580698 13019 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8622650d444fae961f071ed8decfbb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad]), partition=
I20240503 02:12:57.581223 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8622650d444fae961f071ed8decfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.581187 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8622650d444fae961f071ed8decfbb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad]), partition=
I20240503 02:12:57.581704 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8622650d444fae961f071ed8decfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.583781 14104 raft_consensus.cc:979] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: : Instructing follower c6ec2f1283994a3089741715f811c850 to start an election
I20240503 02:12:57.583995 14104 raft_consensus.cc:1067] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Signalling peer c6ec2f1283994a3089741715f811c850 to start an election
I20240503 02:12:57.585788 10491 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78"
dest_uuid: "c6ec2f1283994a3089741715f811c850"
from {username='slave'} at 127.4.5.65:56937
I20240503 02:12:57.586149 10491 raft_consensus.cc:482] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:12:57.586287 10491 raft_consensus.cc:3037] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:57.588958 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 9b8622650d444fae961f071ed8decfbb (DEFAULT_TABLE table=rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad]), partition=
I20240503 02:12:57.589089 10094 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78"
dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:54468
I20240503 02:12:57.589375 10094 raft_consensus.cc:595] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:12:57.589453 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9b8622650d444fae961f071ed8decfbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:57.592193 10491 raft_consensus.cc:504] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:12:57.594111 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:12:57.594472 10357 raft_consensus.cc:3037] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:57.600497 10357 raft_consensus.cc:2454] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 2.
I20240503 02:12:57.604630 10491 leader_election.cc:288] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:57.604650 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:12:57.604997 10093 raft_consensus.cc:3032] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:12:57.605134 10093 raft_consensus.cc:729] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:57.605533 10093 consensus_queue.cc:260] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:12:57.606178 10093 raft_consensus.cc:3037] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:57.608559 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "36457442a8e24ece9a1c2d8cdab89b78" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: true dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:57.608927 12908 raft_consensus.cc:3037] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:12:57.610435 10093 raft_consensus.cc:2454] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 2.
I20240503 02:12:57.611136 10426 leader_election.cc:302] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:57.611631 14079 raft_consensus.cc:2781] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:12:57.611840 14079 raft_consensus.cc:686] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 2 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:57.612244 14079 consensus_queue.cc:237] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:12:57.615121 12908 raft_consensus.cc:2454] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 2.
I20240503 02:12:57.617600 14158 tablet_bootstrap.cc:492] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:57.620635 14159 tablet_bootstrap.cc:492] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:57.621373 14158 tablet_bootstrap.cc:654] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.622637 9944 catalog_manager.cc:5484] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 1 to 2, leader changed from 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 2 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:57.623656 14159 tablet_bootstrap.cc:654] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.625483 14160 tablet_bootstrap.cc:492] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:57.628540 14158 tablet_bootstrap.cc:492] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:57.628779 14158 ts_tablet_manager.cc:1367] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20240503 02:12:57.629606 14160 tablet_bootstrap.cc:654] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:57.629820 14158 raft_consensus.cc:348] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.630133 14158 raft_consensus.cc:374] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.630332 14158 raft_consensus.cc:729] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.630795 14158 consensus_queue.cc:260] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.635519 14158 ts_tablet_manager.cc:1398] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:57.642876 14160 tablet_bootstrap.cc:492] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:57.643110 14160 ts_tablet_manager.cc:1367] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.006s
I20240503 02:12:57.643978 14160 raft_consensus.cc:348] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.644222 14160 raft_consensus.cc:374] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.645130 14160 raft_consensus.cc:729] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.646298 14159 tablet_bootstrap.cc:492] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:57.646560 14159 ts_tablet_manager.cc:1367] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.003s
I20240503 02:12:57.647550 14159 raft_consensus.cc:348] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.647761 14159 raft_consensus.cc:374] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:57.647881 14159 raft_consensus.cc:729] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:57.648285 14159 consensus_queue.cc:260] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.650203 14159 ts_tablet_manager.cc:1398] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:57.650570 14160 consensus_queue.cc:260] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.651474 14160 ts_tablet_manager.cc:1398] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20240503 02:12:57.672698 13943 raft_consensus.cc:482] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:57.672931 13943 raft_consensus.cc:504] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.673717 13943 leader_election.cc:288] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:57.674371 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8622650d444fae961f071ed8decfbb" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:57.674347 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8622650d444fae961f071ed8decfbb" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:57.674629 13040 raft_consensus.cc:2454] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:57.674680 13174 raft_consensus.cc:2454] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:57.675151 12843 leader_election.cc:302] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:57.675433 13943 raft_consensus.cc:2781] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:57.675570 13943 raft_consensus.cc:482] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:57.675670 13943 raft_consensus.cc:3037] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.677934 13943 raft_consensus.cc:504] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.678615 13943 leader_election.cc:288] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:57.678973 13308 raft_consensus.cc:3037] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.679498 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8622650d444fae961f071ed8decfbb" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:57.679513 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9b8622650d444fae961f071ed8decfbb" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:57.679714 13040 raft_consensus.cc:3037] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.679769 13172 raft_consensus.cc:3037] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:57.682873 13172 raft_consensus.cc:2454] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:57.683224 14104 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.684000 12843 leader_election.cc:302] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:57.684583 13943 raft_consensus.cc:2781] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:57.684783 13943 raft_consensus.cc:686] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:12:57.685209 13943 consensus_queue.cc:237] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:57.689491 14104 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.690752 9949 catalog_manager.cc:5484] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:57.698258 13040 raft_consensus.cc:2454] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:57.788316 10093 consensus_queue.cc:237] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:12:57.790578 10491 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:12:57.790613 13040 raft_consensus.cc:1261] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:12:57.791195 14104 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:57.791601 13990 consensus_queue.cc:1035] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:57.794282 13812 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 031a0afb01b6469299fa829fa3467ef7 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.794970 13040 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.801350 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:57.804147 9951 catalog_manager.cc:5484] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 5 to 7, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 2 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.805620 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:57.806687 10491 raft_consensus.cc:2932] T 1fa4eb577d7f42fd8e4a9a4391a0546f P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:12:57.818562 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 1fa4eb577d7f42fd8e4a9a4391a0546f with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42316
I20240503 02:12:57.819255 14099 tablet_replica.cc:330] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:12:57.819520 14099 raft_consensus.cc:2229] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:57.819765 14099 raft_consensus.cc:2258] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:57.821449 14099 ts_tablet_manager.cc:1875] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:57.822180 13172 raft_consensus.cc:1261] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:57.823029 13805 consensus_queue.cc:1035] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.826925 13040 raft_consensus.cc:1261] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:57.827827 13805 consensus_queue.cc:1035] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.831251 14099 ts_tablet_manager.cc:1888] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:57.831419 14099 log.cc:1196] T 1fa4eb577d7f42fd8e4a9a4391a0546f P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/1fa4eb577d7f42fd8e4a9a4391a0546f
I20240503 02:12:57.833954 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 1fa4eb577d7f42fd8e4a9a4391a0546f (table rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]) successfully deleted
I20240503 02:12:57.848380 14176 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:57.850064 14176 tablet.cc:1620] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:57.851346 9949 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 9b8622650d444fae961f071ed8decfbb (table rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad]). Expected version 1 got 0
I20240503 02:12:57.860265 14175 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:57.863902 14175 tablet.cc:1620] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:57.868518 9951 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 9b8622650d444fae961f071ed8decfbb (table rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad]). Expected version 1 got 0
I20240503 02:12:57.869716 9946 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 9b8622650d444fae961f071ed8decfbb (table rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad]). Expected version 1 got 0
I20240503 02:12:57.879566 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad] alter complete (version 1)
I20240503 02:12:57.881168 14178 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:57.882112 14178 tablet.cc:1620] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:12:57.889268 9946 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 9b8622650d444fae961f071ed8decfbb (table rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad]). Expected version 1 got 0
I20240503 02:12:57.984845 13943 consensus_queue.cc:1035] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.990962 13805 consensus_queue.cc:1035] T bf4870715de449cda20ad5f2e02397c9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:57.991542 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:57.999622 10093 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:58.000833 13487 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:58.009868 14175 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.010560 14175 tablet.cc:1620] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:58.010829 14178 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.011408 14178 tablet.cc:1620] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:12:58.012531 14176 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.013026 14176 tablet.cc:1620] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:58.020304 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=e38f3b6e2aae40ce8a0b054945a0bdad] alter complete (version 2)
I20240503 02:12:58.082473 12908 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:58.083257 13487 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:58.090458 10357 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:12:58.090785 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:58.091203 13487 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:12:58.103633 13153 tablet_service.cc:1467] Processing CreateTablet for tablet e3bba75394f24d4c91562089abad24c4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.103979 13019 tablet_service.cc:1467] Processing CreateTablet for tablet e3bba75394f24d4c91562089abad24c4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.104257 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3bba75394f24d4c91562089abad24c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.104475 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3bba75394f24d4c91562089abad24c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.107456 13288 tablet_service.cc:1467] Processing CreateTablet for tablet e3bba75394f24d4c91562089abad24c4 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.108019 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3bba75394f24d4c91562089abad24c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.113080 13018 tablet_service.cc:1467] Processing CreateTablet for tablet db2da39193164d1ebf830282258d47d7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.113580 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2da39193164d1ebf830282258d47d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.114125 13152 tablet_service.cc:1467] Processing CreateTablet for tablet e5968b9989fb4f37bb4422645e7c8032 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.114650 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5968b9989fb4f37bb4422645e7c8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.114125 13154 tablet_service.cc:1467] Processing CreateTablet for tablet db2da39193164d1ebf830282258d47d7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.115628 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2da39193164d1ebf830282258d47d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.116067 12887 tablet_service.cc:1467] Processing CreateTablet for tablet e5968b9989fb4f37bb4422645e7c8032 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.116573 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5968b9989fb4f37bb4422645e7c8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.113605 14159 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:58.116649 12888 tablet_service.cc:1467] Processing CreateTablet for tablet db2da39193164d1ebf830282258d47d7 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.117029 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2da39193164d1ebf830282258d47d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.118615 14159 tablet_bootstrap.cc:654] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.112988 13020 tablet_service.cc:1467] Processing CreateTablet for tablet e5968b9989fb4f37bb4422645e7c8032 (DEFAULT_TABLE table=rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:58.121181 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5968b9989fb4f37bb4422645e7c8032. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.123211 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:58.135566 14158 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:58.137641 14158 tablet_bootstrap.cc:654] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.141383 14160 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:58.142256 14203 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:58.143273 14160 tablet_bootstrap.cc:654] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.144181 14203 tablet_bootstrap.cc:654] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.149279 14159 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:58.151177 14159 ts_tablet_manager.cc:1367] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.002s
I20240503 02:12:58.152495 14159 raft_consensus.cc:348] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.153035 14159 raft_consensus.cc:374] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.152933 14175 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.153205 14159 raft_consensus.cc:729] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.153574 14159 consensus_queue.cc:260] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.154553 14176 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.156149 14159 ts_tablet_manager.cc:1398] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20240503 02:12:58.157045 14159 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:58.162425 14159 tablet_bootstrap.cc:654] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.165705 14158 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:58.165778 14203 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:58.165956 14203 ts_tablet_manager.cc:1367] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.012s
I20240503 02:12:58.166093 14158 ts_tablet_manager.cc:1367] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240503 02:12:58.166960 14158 raft_consensus.cc:348] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.166962 14203 raft_consensus.cc:348] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.167222 14203 raft_consensus.cc:374] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.167313 14203 raft_consensus.cc:729] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.167372 14158 raft_consensus.cc:374] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.167551 14158 raft_consensus.cc:729] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.167634 14203 consensus_queue.cc:260] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.168036 14158 consensus_queue.cc:260] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.168757 14203 ts_tablet_manager.cc:1398] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20240503 02:12:58.172101 14158 ts_tablet_manager.cc:1398] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:12:58.172546 14158 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:58.173449 14159 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:58.173633 14159 ts_tablet_manager.cc:1367] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.001s
I20240503 02:12:58.174633 14159 raft_consensus.cc:348] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.174894 14159 raft_consensus.cc:374] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.174997 14159 raft_consensus.cc:729] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.175432 14159 consensus_queue.cc:260] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.176314 14159 ts_tablet_manager.cc:1398] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:58.176645 14159 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:58.177146 14158 tablet_bootstrap.cc:654] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.178490 14159 tablet_bootstrap.cc:654] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.181622 14160 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:58.181897 14160 ts_tablet_manager.cc:1367] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.008s
I20240503 02:12:58.182952 14160 raft_consensus.cc:348] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.183323 14160 raft_consensus.cc:374] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.183470 14160 raft_consensus.cc:729] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.183900 14160 consensus_queue.cc:260] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.186043 14160 ts_tablet_manager.cc:1398] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:12:58.186424 14160 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:58.188385 14160 tablet_bootstrap.cc:654] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.199250 14160 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:58.199609 14160 ts_tablet_manager.cc:1367] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.010s
I20240503 02:12:58.200721 14160 raft_consensus.cc:348] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.201043 14160 raft_consensus.cc:374] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.201256 14160 raft_consensus.cc:729] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.201612 14160 consensus_queue.cc:260] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.202378 14160 ts_tablet_manager.cc:1398] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:58.203161 14178 tablet.cc:1722] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:58.207399 13890 raft_consensus.cc:482] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.207625 13890 raft_consensus.cc:504] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.208518 13890 leader_election.cc:288] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:58.209829 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:58.209923 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:58.210110 13308 raft_consensus.cc:2454] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:58.210719 12973 leader_election.cc:302] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:12:58.211120 14219 raft_consensus.cc:2781] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
W20240503 02:12:58.211355 12976 leader_election.cc:341] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:58.211477 14219 raft_consensus.cc:482] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:58.211607 14219 raft_consensus.cc:3037] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.215720 14219 raft_consensus.cc:504] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.216683 14219 leader_election.cc:288] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:58.217355 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
W20240503 02:12:58.218066 12976 leader_election.cc:341] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:58.218196 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:58.218447 13308 raft_consensus.cc:3037] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.221443 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=e38f3b6e2aae40ce8a0b054945a0bdad] alter complete (version 3)
I20240503 02:12:58.226406 13308 raft_consensus.cc:2454] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:58.227615 12973 leader_election.cc:302] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: f93b103af083471b8e8c0376e9148aad
I20240503 02:12:58.229511 14219 raft_consensus.cc:2781] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
W20240503 02:12:58.285434 10294 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.65:38119, user_credentials={real_user=slave}} blocked reactor thread for 108939us
I20240503 02:12:58.303498 13890 raft_consensus.cc:482] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.303773 13890 raft_consensus.cc:504] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.304705 13890 leader_election.cc:288] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:58.306695 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:58.307046 12908 raft_consensus.cc:2454] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:58.307826 12973 leader_election.cc:302] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:58.308262 13890 raft_consensus.cc:2781] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:58.308418 13890 raft_consensus.cc:482] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:58.308557 13890 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.312914 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
W20240503 02:12:58.313925 12976 leader_election.cc:341] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:58.316699 13890 raft_consensus.cc:504] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.318892 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:58.319171 12908 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.323958 12908 raft_consensus.cc:2454] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:58.324867 12973 leader_election.cc:302] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:58.325825 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
W20240503 02:12:58.329452 12976 leader_election.cc:341] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:58.330700 13890 leader_election.cc:288] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:58.331291 13890 raft_consensus.cc:2781] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:58.331539 13890 raft_consensus.cc:686] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:58.332055 13890 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.338063 9952 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:58.338560 9952 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:58.346877 9952 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 9b8622650d444fae961f071ed8decfbb
I20240503 02:12:58.349033 13154 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b8622650d444fae961f071ed8decfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:12:58.374133 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b8622650d444fae961f071ed8decfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:12:58.398898 14227 tablet_replica.cc:330] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:12:58.400118 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 9b8622650d444fae961f071ed8decfbb with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:58 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:12:58.400415 14227 raft_consensus.cc:2229] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:58.401101 14227 raft_consensus.cc:2258] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.402925 14227 ts_tablet_manager.cc:1875] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.403621 14228 tablet_replica.cc:330] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:12:58.403864 14228 raft_consensus.cc:2229] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.404189 14228 raft_consensus.cc:2258] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.409364 14228 ts_tablet_manager.cc:1875] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.410367 14158 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:58.410764 14158 ts_tablet_manager.cc:1367] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.238s user 0.040s sys 0.103s
I20240503 02:12:58.412431 14158 raft_consensus.cc:348] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.412891 14158 raft_consensus.cc:374] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.413091 14158 raft_consensus.cc:729] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.413542 14158 consensus_queue.cc:260] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.414299 14158 ts_tablet_manager.cc:1398] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:12:58.415264 14158 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:58.417548 14158 tablet_bootstrap.cc:654] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.421653 14224 tablet_replica.cc:330] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:12:58.421989 14224 raft_consensus.cc:2229] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.423485 14224 raft_consensus.cc:2258] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.425601 14224 ts_tablet_manager.cc:1875] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:58.430814 14219 raft_consensus.cc:686] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:58.431327 14219 consensus_queue.cc:237] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.430740 14158 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:58.432404 14158 ts_tablet_manager.cc:1367] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.002s
I20240503 02:12:58.432155 14159 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:58.433547 14159 ts_tablet_manager.cc:1367] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.257s user 0.069s sys 0.119s
I20240503 02:12:58.433458 14158 raft_consensus.cc:348] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.433944 14158 raft_consensus.cc:374] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.434082 14158 raft_consensus.cc:729] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.434412 14158 consensus_queue.cc:260] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.434548 14159 raft_consensus.cc:348] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.434782 14159 raft_consensus.cc:374] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.434871 14159 raft_consensus.cc:729] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.435134 14159 consensus_queue.cc:260] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.435977 14159 ts_tablet_manager.cc:1398] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:12:58.438221 14158 ts_tablet_manager.cc:1398] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:12:58.439090 14228 ts_tablet_manager.cc:1888] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.439234 14228 log.cc:1196] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/9b8622650d444fae961f071ed8decfbb
I20240503 02:12:58.439110 9952 catalog_manager.cc:5484] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:58.439769 14228 ts_tablet_manager.cc:1909] T 9b8622650d444fae961f071ed8decfbb P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:12:58.441022 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 9b8622650d444fae961f071ed8decfbb (table rebalancer_dynamic_table_new_name [id=e38f3b6e2aae40ce8a0b054945a0bdad]) successfully deleted
I20240503 02:12:58.444502 14227 ts_tablet_manager.cc:1888] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.444761 14227 log.cc:1196] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/9b8622650d444fae961f071ed8decfbb
I20240503 02:12:58.445371 14227 ts_tablet_manager.cc:1909] T 9b8622650d444fae961f071ed8decfbb P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:12:58.447427 14224 ts_tablet_manager.cc:1888] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:58.447582 14224 log.cc:1196] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/9b8622650d444fae961f071ed8decfbb
I20240503 02:12:58.449128 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 9b8622650d444fae961f071ed8decfbb (table rebalancer_dynamic_table_new_name [id=e38f3b6e2aae40ce8a0b054945a0bdad]) successfully deleted
I20240503 02:12:58.449486 14224 ts_tablet_manager.cc:1909] T 9b8622650d444fae961f071ed8decfbb P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:12:58.450145 9952 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:58.455415 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 9b8622650d444fae961f071ed8decfbb (table rebalancer_dynamic_table_new_name [id=e38f3b6e2aae40ce8a0b054945a0bdad]) successfully deleted
I20240503 02:12:58.477376 14219 raft_consensus.cc:482] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.477586 14219 raft_consensus.cc:504] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.478549 14219 leader_election.cc:288] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:58.479084 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5968b9989fb4f37bb4422645e7c8032" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:58.479342 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5968b9989fb4f37bb4422645e7c8032" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:58.479434 13172 raft_consensus.cc:2454] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:58.479657 12908 raft_consensus.cc:2454] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:12:58.479907 12976 leader_election.cc:302] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:58.480540 14219 raft_consensus.cc:2781] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:58.480687 14219 raft_consensus.cc:482] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:58.480790 14219 raft_consensus.cc:3037] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.483383 14219 raft_consensus.cc:504] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.484143 14219 leader_election.cc:288] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:58.484879 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5968b9989fb4f37bb4422645e7c8032" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:58.484880 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e5968b9989fb4f37bb4422645e7c8032" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:58.485162 13172 raft_consensus.cc:3037] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.485162 12908 raft_consensus.cc:3037] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.488459 13172 raft_consensus.cc:2454] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:58.489266 12976 leader_election.cc:302] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:58.489737 12908 raft_consensus.cc:2454] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:12:58.489641 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:58.490567 14219 raft_consensus.cc:2781] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:58.490839 14219 raft_consensus.cc:686] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:12:58.491194 14219 consensus_queue.cc:237] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.495618 9944 catalog_manager.cc:5484] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:58.508402 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 15197307da66490d9dd754e7e0c4bd06 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980]), partition=
I20240503 02:12:58.508986 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15197307da66490d9dd754e7e0c4bd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.509164 10491 consensus_queue.cc:237] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:12:58.510924 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 15197307da66490d9dd754e7e0c4bd06 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980]), partition=
I20240503 02:12:58.511365 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15197307da66490d9dd754e7e0c4bd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.517282 14159 tablet_bootstrap.cc:492] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:58.517320 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 15197307da66490d9dd754e7e0c4bd06 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980]), partition=
I20240503 02:12:58.517724 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15197307da66490d9dd754e7e0c4bd06. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:58.519037 14159 tablet_bootstrap.cc:654] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.520607 10359 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:12:58.520978 12908 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:12:58.521770 14075 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:58.522094 14159 tablet_bootstrap.cc:492] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:58.522257 14159 ts_tablet_manager.cc:1367] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.003s
I20240503 02:12:58.522429 14075 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:12:58.525174 14079 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:12:58.525537 14159 raft_consensus.cc:348] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.525759 14159 raft_consensus.cc:374] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.525864 14159 raft_consensus.cc:729] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.526917 14159 consensus_queue.cc:260] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.531857 14160 tablet_bootstrap.cc:492] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:58.534003 14160 tablet_bootstrap.cc:654] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.534313 12908 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:12:58.534988 14159 ts_tablet_manager.cc:1398] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240503 02:12:58.536404 10359 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:12:58.542763 9951 catalog_manager.cc:5484] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 5 to 7, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 2 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:12:58.548013 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 36457442a8e24ece9a1c2d8cdab89b78 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:12:58.549145 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 36457442a8e24ece9a1c2d8cdab89b78 with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:36690
I20240503 02:12:58.553062 14234 tablet_replica.cc:330] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:12:58.554612 14160 tablet_bootstrap.cc:492] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:58.554924 14160 ts_tablet_manager.cc:1367] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.001s
I20240503 02:12:58.555846 14160 raft_consensus.cc:348] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.556121 14160 raft_consensus.cc:374] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.556212 14160 raft_consensus.cc:729] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.556499 14160 consensus_queue.cc:260] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.556764 14234 raft_consensus.cc:2229] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:58.557274 14160 ts_tablet_manager.cc:1398] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:12:58.557298 14234 raft_consensus.cc:2258] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:58.559263 14234 ts_tablet_manager.cc:1875] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:12:58.567862 14235 tablet_bootstrap.cc:492] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:12:58.569959 14196 raft_consensus.cc:482] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.570155 14196 raft_consensus.cc:504] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:12:58.571184 14196 leader_election.cc:288] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:58.571208 14234 ts_tablet_manager.cc:1888] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.6
I20240503 02:12:58.571350 14234 log.cc:1196] T 36457442a8e24ece9a1c2d8cdab89b78 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/36457442a8e24ece9a1c2d8cdab89b78
I20240503 02:12:58.573678 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 36457442a8e24ece9a1c2d8cdab89b78 (table rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]) successfully deleted
I20240503 02:12:58.575486 14235 tablet_bootstrap.cc:654] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:58.577334 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:58.577800 12908 raft_consensus.cc:2379] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f93b103af083471b8e8c0376e9148aad in current term 1: Already voted for candidate 8d454a3607d042dc9559ed59c3ccd84b in this term.
I20240503 02:12:58.585098 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:58.585817 14235 tablet_bootstrap.cc:492] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:12:58.585840 13110 leader_election.cc:302] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:58.585994 14235 ts_tablet_manager.cc:1367] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.012s
I20240503 02:12:58.586210 14196 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.587033 14235 raft_consensus.cc:348] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.587285 14235 raft_consensus.cc:374] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:58.587399 14235 raft_consensus.cc:729] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:12:58.587647 14235 consensus_queue.cc:260] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.588325 14235 ts_tablet_manager.cc:1398] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:12:58.589720 14196 raft_consensus.cc:2726] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:58.598865 14079 raft_consensus.cc:482] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.599099 14079 raft_consensus.cc:504] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.599836 14079 leader_election.cc:288] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:58.600497 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15197307da66490d9dd754e7e0c4bd06" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:58.600538 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15197307da66490d9dd754e7e0c4bd06" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:58.600806 13040 raft_consensus.cc:2454] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:58.600865 12908 raft_consensus.cc:2454] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:12:58.601354 10427 leader_election.cc:302] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:12:58.601655 14079 raft_consensus.cc:2781] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:58.601747 14079 raft_consensus.cc:482] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:58.601866 14079 raft_consensus.cc:3037] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.604097 14079 raft_consensus.cc:504] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.604940 14079 leader_election.cc:288] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:12:58.605412 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15197307da66490d9dd754e7e0c4bd06" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:58.605576 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15197307da66490d9dd754e7e0c4bd06" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:12:58.605827 12908 raft_consensus.cc:3037] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.606609 13040 raft_consensus.cc:3037] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.609225 12908 raft_consensus.cc:2454] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:58.609858 10424 leader_election.cc:302] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:58.610406 14079 raft_consensus.cc:2781] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:58.610637 14079 raft_consensus.cc:686] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:12:58.611037 14079 consensus_queue.cc:237] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:12:58.614883 13040 raft_consensus.cc:2454] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:12:58.616391 9952 catalog_manager.cc:5484] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:58.742270 9952 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:58.753964 12908 raft_consensus.cc:1261] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:58.754102 13040 raft_consensus.cc:1261] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:58.754868 14075 consensus_queue.cc:1035] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.755425 14075 consensus_queue.cc:1035] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.764050 9952 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 15197307da66490d9dd754e7e0c4bd06 (table rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980]). Expected version 1 got 0
I20240503 02:12:58.769295 14276 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:58.770093 14276 tablet.cc:1620] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:12:58.770776 9952 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet 15197307da66490d9dd754e7e0c4bd06 (table rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980]). Expected version 1 got 0
I20240503 02:12:58.775388 14277 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:58.776147 14277 tablet.cc:1620] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:58.779688 9951 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 15197307da66490d9dd754e7e0c4bd06 (table rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980]). Expected version 1 got 0
I20240503 02:12:58.791846 9946 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 15197307da66490d9dd754e7e0c4bd06 (table rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980]). Expected version 1 got 0
I20240503 02:12:58.795374 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980] alter complete (version 1)
I20240503 02:12:58.814235 14278 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:58.815557 14278 tablet.cc:1620] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:58.818498 9951 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 15197307da66490d9dd754e7e0c4bd06 (table rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980]). Expected version 1 got 0
I20240503 02:12:58.860323 14219 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.879614 14219 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.880280 14196 raft_consensus.cc:482] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:58.880517 14196 raft_consensus.cc:504] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:12:58.881330 14196 leader_election.cc:288] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:12:58.881882 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:58.881878 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:58.882769 13107 leader_election.cc:302] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:12:58.883301 14196 raft_consensus.cc:2726] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:58.892275 14220 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.892865 14220 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.902684 14220 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.908363 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:12:58.921192 13174 raft_consensus.cc:3037] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:58.929105 13890 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:58.941121 14276 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.941792 14276 tablet.cc:1620] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:12:58.942597 14278 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.943408 14278 tablet.cc:1620] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:58.954334 14277 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:12:58.954979 14277 tablet.cc:1620] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:58.970595 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=4b5380bebfad430abbb8618e7a3bf980] alter complete (version 2)
I20240503 02:12:59.021767 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_16"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:12:59.038908 10072 tablet_service.cc:1467] Processing CreateTablet for tablet e4caac5d5607431ba8af43993e9de075 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.039662 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4caac5d5607431ba8af43993e9de075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.051591 13020 tablet_service.cc:1467] Processing CreateTablet for tablet e4caac5d5607431ba8af43993e9de075 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.051726 13154 tablet_service.cc:1467] Processing CreateTablet for tablet e4caac5d5607431ba8af43993e9de075 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.052223 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4caac5d5607431ba8af43993e9de075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.052212 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4caac5d5607431ba8af43993e9de075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.057732 12887 tablet_service.cc:1467] Processing CreateTablet for tablet d856c707315b47e996ee604a5ff66b31 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.058413 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d856c707315b47e996ee604a5ff66b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.062206 13288 tablet_service.cc:1467] Processing CreateTablet for tablet b33490610e3b4819a61ef4bbfe0c5942 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.062932 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33490610e3b4819a61ef4bbfe0c5942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.063557 13153 tablet_service.cc:1467] Processing CreateTablet for tablet d856c707315b47e996ee604a5ff66b31 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.064127 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d856c707315b47e996ee604a5ff66b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.068006 13152 tablet_service.cc:1467] Processing CreateTablet for tablet b33490610e3b4819a61ef4bbfe0c5942 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.069645 13018 tablet_service.cc:1467] Processing CreateTablet for tablet d856c707315b47e996ee604a5ff66b31 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.070132 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d856c707315b47e996ee604a5ff66b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.074726 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33490610e3b4819a61ef4bbfe0c5942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.076932 9949 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:12:59.103312 14276 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:59.106122 14308 tablet_bootstrap.cc:492] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:59.108927 14308 tablet_bootstrap.cc:654] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.113298 14277 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:59.122774 14278 tablet.cc:1722] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:12:59.123214 14308 tablet_bootstrap.cc:492] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:59.123129 14309 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:12:59.123409 14308 ts_tablet_manager.cc:1367] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20240503 02:12:59.124372 14308 raft_consensus.cc:348] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.127177 14308 raft_consensus.cc:374] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.127583 14308 raft_consensus.cc:729] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.128542 14308 consensus_queue.cc:260] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.131619 14308 ts_tablet_manager.cc:1398] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.008s user 0.002s sys 0.002s
I20240503 02:12:59.142553 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=4b5380bebfad430abbb8618e7a3bf980] alter complete (version 3)
I20240503 02:12:59.127041 14309 tablet_bootstrap.cc:654] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.169515 14197 raft_consensus.cc:482] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.169771 14197 raft_consensus.cc:504] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.170838 14197 leader_election.cc:288] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
W20240503 02:12:59.174160 13246 leader_election.cc:341] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Not found: Tablet not found: b33490610e3b4819a61ef4bbfe0c5942
I20240503 02:12:59.196187 14315 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:59.200222 14316 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:59.208837 14316 tablet_bootstrap.cc:654] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.231498 14309 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:12:59.234268 14309 ts_tablet_manager.cc:1367] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.112s user 0.012s sys 0.012s
I20240503 02:12:59.245791 9949 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:59.247115 9949 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:12:59.249570 13288 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:59.254849 9949 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 15197307da66490d9dd754e7e0c4bd06
I20240503 02:12:59.255813 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 15197307da66490d9dd754e7e0c4bd06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:12:59.256026 10337 tablet_service.cc:1467] Processing CreateTablet for tablet b33490610e3b4819a61ef4bbfe0c5942 (DEFAULT_TABLE table=rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:12:59.256601 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b33490610e3b4819a61ef4bbfe0c5942. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.256709 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 15197307da66490d9dd754e7e0c4bd06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:12:59.238366 14309 raft_consensus.cc:348] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.262578 10072 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:12:59.262112 14309 raft_consensus.cc:374] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.263324 14309 raft_consensus.cc:729] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.263718 14309 consensus_queue.cc:260] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.272689 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 15197307da66490d9dd754e7e0c4bd06 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:12:59 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:12:59.273129 14309 ts_tablet_manager.cc:1398] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.038s user 0.001s sys 0.001s
I20240503 02:12:59.280984 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33490610e3b4819a61ef4bbfe0c5942" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
W20240503 02:12:59.282822 13245 leader_election.cc:341] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:59.283043 13245 leader_election.cc:302] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f, f93b103af083471b8e8c0376e9148aad
I20240503 02:12:59.284266 14197 raft_consensus.cc:2726] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:59.286756 14315 tablet_bootstrap.cc:654] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.290692 14328 tablet_replica.cc:330] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:12:59.291152 14328 raft_consensus.cc:2229] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:12:59.291793 14328 raft_consensus.cc:2258] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.292932 14328 ts_tablet_manager.cc:1875] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.289903 13152 tablet_service.cc:1429] Tablet server has 2 leaders and 0 scanners
I20240503 02:12:59.295677 14327 tablet_replica.cc:330] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:12:59.296046 14327 raft_consensus.cc:2229] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.296386 14327 raft_consensus.cc:2258] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.299234 14327 ts_tablet_manager.cc:1875] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.310879 14331 tablet_replica.cc:330] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:12:59.311395 14331 raft_consensus.cc:2229] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:12:59.311940 14331 raft_consensus.cc:2258] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:12:59.314060 14328 ts_tablet_manager.cc:1888] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.314373 14328 log.cc:1196] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/15197307da66490d9dd754e7e0c4bd06
I20240503 02:12:59.315587 14331 ts_tablet_manager.cc:1875] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:12:59.318562 10469 tablet_service.cc:1429] Tablet server has 7 leaders and 0 scanners
I20240503 02:12:59.321962 14330 tablet_bootstrap.cc:492] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:12:59.327550 14307 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:59.330372 10337 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:12:59.333398 14327 ts_tablet_manager.cc:1888] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.334158 14327 log.cc:1196] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/15197307da66490d9dd754e7e0c4bd06
I20240503 02:12:59.334817 14327 ts_tablet_manager.cc:1909] T 15197307da66490d9dd754e7e0c4bd06 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:12:59.335001 13018 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:12:59.338233 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 15197307da66490d9dd754e7e0c4bd06 (table rebalancer_dynamic_table_new_name [id=4b5380bebfad430abbb8618e7a3bf980]) successfully deleted
I20240503 02:12:59.341568 14307 tablet_bootstrap.cc:654] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.346297 14328 ts_tablet_manager.cc:1909] T 15197307da66490d9dd754e7e0c4bd06 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:12:59.347529 12887 tablet_service.cc:1429] Tablet server has 4 leaders and 0 scanners
I20240503 02:12:59.348842 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 15197307da66490d9dd754e7e0c4bd06 (table rebalancer_dynamic_table_new_name [id=4b5380bebfad430abbb8618e7a3bf980]) successfully deleted
I20240503 02:12:59.360146 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:12:59.378335 10206 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:12:59.379166 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c]), partition=
I20240503 02:12:59.379165 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c]), partition=
I20240503 02:12:59.379643 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.379643 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.383625 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c]), partition=
I20240503 02:12:59.384177 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:12:59.396409 14307 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:59.398998 14307 ts_tablet_manager.cc:1367] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.001s
I20240503 02:12:59.402127 14307 raft_consensus.cc:348] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.402431 14307 raft_consensus.cc:374] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.402571 14307 raft_consensus.cc:729] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.402899 14307 consensus_queue.cc:260] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.407727 14331 ts_tablet_manager.cc:1888] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:12:59.411617 14331 log.cc:1196] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/15197307da66490d9dd754e7e0c4bd06
I20240503 02:12:59.413708 14307 ts_tablet_manager.cc:1398] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20240503 02:12:59.414124 14308 tablet_bootstrap.cc:492] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:12:59.414327 14307 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:59.416119 14308 tablet_bootstrap.cc:654] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.416118 14307 tablet_bootstrap.cc:654] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.422403 14331 ts_tablet_manager.cc:1909] T 15197307da66490d9dd754e7e0c4bd06 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:12:59.424538 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 15197307da66490d9dd754e7e0c4bd06 (table rebalancer_dynamic_table_new_name [id=4b5380bebfad430abbb8618e7a3bf980]) successfully deleted
I20240503 02:12:59.425211 14308 tablet_bootstrap.cc:492] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:12:59.425433 14308 ts_tablet_manager.cc:1367] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.004s
I20240503 02:12:59.427084 14308 raft_consensus.cc:348] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.428824 14308 raft_consensus.cc:374] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.428989 14308 raft_consensus.cc:729] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.430821 14308 consensus_queue.cc:260] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.433035 14308 ts_tablet_manager.cc:1398] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20240503 02:12:59.454027 14315 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:59.454325 14315 ts_tablet_manager.cc:1367] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.258s user 0.007s sys 0.000s
I20240503 02:12:59.455585 14315 raft_consensus.cc:348] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.455886 14315 raft_consensus.cc:374] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.455987 14315 raft_consensus.cc:729] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.456524 14315 consensus_queue.cc:260] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.458431 14315 ts_tablet_manager.cc:1398] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:12:59.458874 14315 tablet_bootstrap.cc:492] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:12:59.460354 14330 tablet_bootstrap.cc:654] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.465550 14315 tablet_bootstrap.cc:654] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.473978 14197 raft_consensus.cc:482] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.474200 14197 raft_consensus.cc:504] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.474969 14197 leader_election.cc:288] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:59.477134 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a8ca5ec12f4e2baa9d5b5f7d0f00e8" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:59.477360 14330 tablet_bootstrap.cc:492] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:12:59.477559 14330 ts_tablet_manager.cc:1367] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.156s user 0.005s sys 0.000s
I20240503 02:12:59.478888 14330 raft_consensus.cc:348] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.479164 14330 raft_consensus.cc:374] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.479293 14330 raft_consensus.cc:729] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
W20240503 02:12:59.479555 13246 leader_election.cc:341] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:59.479626 14330 consensus_queue.cc:260] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.480155 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a8ca5ec12f4e2baa9d5b5f7d0f00e8" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:12:59.480397 14330 ts_tablet_manager.cc:1398] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20240503 02:12:59.482635 13243 leader_election.cc:341] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:12:59.483212 13243 leader_election.cc:302] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:12:59.483794 14197 raft_consensus.cc:2726] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:12:59.495992 14315 tablet_bootstrap.cc:492] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:12:59.496256 14315 ts_tablet_manager.cc:1367] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.000s
I20240503 02:12:59.497313 14315 raft_consensus.cc:348] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.505295 14316 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:59.505793 14316 ts_tablet_manager.cc:1367] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.306s user 0.013s sys 0.012s
I20240503 02:12:59.506964 14316 raft_consensus.cc:348] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.514237 14315 raft_consensus.cc:374] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.514577 14315 raft_consensus.cc:729] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.515151 14315 consensus_queue.cc:260] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.517446 14315 ts_tablet_manager.cc:1398] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20240503 02:12:59.530464 14307 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:59.530740 14316 raft_consensus.cc:374] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.530985 14316 raft_consensus.cc:729] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.531348 14316 consensus_queue.cc:260] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.533282 14316 ts_tablet_manager.cc:1398] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20240503 02:12:59.533722 14316 tablet_bootstrap.cc:492] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:59.535843 14316 tablet_bootstrap.cc:654] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.536260 14307 ts_tablet_manager.cc:1367] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.122s user 0.001s sys 0.011s
I20240503 02:12:59.543201 14104 raft_consensus.cc:482] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.543444 14104 raft_consensus.cc:504] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.544003 14307 raft_consensus.cc:348] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.544215 14104 leader_election.cc:288] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
W20240503 02:12:59.544791 13107 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.69:33471, user_credentials={real_user=slave}} blocked reactor thread for 73356.5us
I20240503 02:12:59.545445 14307 raft_consensus.cc:374] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.545706 14307 raft_consensus.cc:729] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.546329 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4caac5d5607431ba8af43993e9de075" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:59.546547 13174 raft_consensus.cc:2454] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:59.547003 14307 consensus_queue.cc:260] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.550848 14307 ts_tablet_manager.cc:1398] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20240503 02:12:59.551314 14307 tablet_bootstrap.cc:492] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:12:59.553267 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4caac5d5607431ba8af43993e9de075" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:59.554847 10029 leader_election.cc:302] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:59.555243 14104 raft_consensus.cc:2781] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.555341 14104 raft_consensus.cc:482] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.555442 14104 raft_consensus.cc:3037] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.558910 14104 raft_consensus.cc:504] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.563282 14104 leader_election.cc:288] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:59.564004 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4caac5d5607431ba8af43993e9de075" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:59.564303 13174 raft_consensus.cc:3037] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.565567 14316 tablet_bootstrap.cc:492] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:59.565755 14316 ts_tablet_manager.cc:1367] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.001s
I20240503 02:12:59.566854 14316 raft_consensus.cc:348] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.567133 14316 raft_consensus.cc:374] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.567236 14316 raft_consensus.cc:729] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.567523 14316 consensus_queue.cc:260] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.569257 14316 ts_tablet_manager.cc:1398] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:12:59.569428 13040 raft_consensus.cc:2454] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:12:59.570745 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e4caac5d5607431ba8af43993e9de075" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:59.571333 14316 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:12:59.571314 13040 raft_consensus.cc:3037] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.577759 14316 tablet_bootstrap.cc:654] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.578533 13174 raft_consensus.cc:2454] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:59.581470 10029 leader_election.cc:302] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:59.581920 14104 raft_consensus.cc:2781] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.582096 14104 raft_consensus.cc:686] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:12:59.582437 14104 consensus_queue.cc:237] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.601109 13040 raft_consensus.cc:2454] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:12:59.613526 14307 tablet_bootstrap.cc:654] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:12:59.616798 9952 catalog_manager.cc:5484] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:59.625869 14316 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:12:59.626225 14316 ts_tablet_manager.cc:1367] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.010s
I20240503 02:12:59.627161 14316 raft_consensus.cc:348] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.627493 14316 raft_consensus.cc:374] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.627624 14316 raft_consensus.cc:729] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.627941 14316 consensus_queue.cc:260] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.629587 14316 ts_tablet_manager.cc:1398] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:12:59.636377 14307 tablet_bootstrap.cc:492] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:12:59.637712 14307 ts_tablet_manager.cc:1367] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.085s user 0.005s sys 0.000s
I20240503 02:12:59.638909 14307 raft_consensus.cc:348] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.639206 14307 raft_consensus.cc:374] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:12:59.639310 14307 raft_consensus.cc:729] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:12:59.639643 14307 consensus_queue.cc:260] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.640414 14307 ts_tablet_manager.cc:1398] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.003s user 0.000s sys 0.000s
I20240503 02:12:59.679381 13805 raft_consensus.cc:482] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.679654 13805 raft_consensus.cc:504] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.680464 13805 leader_election.cc:288] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:59.681174 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a8ca5ec12f4e2baa9d5b5f7d0f00e8" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:59.681473 13040 raft_consensus.cc:2454] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:59.681437 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a8ca5ec12f4e2baa9d5b5f7d0f00e8" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:59.681746 13308 raft_consensus.cc:2454] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:59.682296 12843 leader_election.cc:302] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:59.682685 13805 raft_consensus.cc:2781] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.682821 13805 raft_consensus.cc:482] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.682937 13805 raft_consensus.cc:3037] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.686460 13805 raft_consensus.cc:504] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.687111 13805 leader_election.cc:288] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:12:59.687597 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a8ca5ec12f4e2baa9d5b5f7d0f00e8" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:59.687836 13040 raft_consensus.cc:3037] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.688659 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "40a8ca5ec12f4e2baa9d5b5f7d0f00e8" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:59.688921 13308 raft_consensus.cc:3037] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.692494 13308 raft_consensus.cc:2454] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:59.693156 12840 leader_election.cc:302] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:12:59.693265 14223 raft_consensus.cc:482] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.693485 14223 raft_consensus.cc:504] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.694613 13805 raft_consensus.cc:2781] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.694661 14223 leader_election.cc:288] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:59.700696 13805 raft_consensus.cc:686] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:12:59.701053 13805 consensus_queue.cc:237] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:12:59.701449 13040 raft_consensus.cc:2454] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:59.712013 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33490610e3b4819a61ef4bbfe0c5942" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:59.712427 13174 raft_consensus.cc:2454] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:12:59.713065 10295 leader_election.cc:302] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:59.713421 14223 raft_consensus.cc:2781] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.713596 14223 raft_consensus.cc:482] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.713702 14223 raft_consensus.cc:3037] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.717788 14223 raft_consensus.cc:504] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.717779 9952 catalog_manager.cc:5484] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:59.718788 14223 leader_election.cc:288] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:59.720613 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33490610e3b4819a61ef4bbfe0c5942" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:59.720896 13174 raft_consensus.cc:3037] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.724048 13174 raft_consensus.cc:2454] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:59.725000 10295 leader_election.cc:302] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:59.725482 14223 raft_consensus.cc:2781] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.725849 14223 raft_consensus.cc:686] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:12:59.725852 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33490610e3b4819a61ef4bbfe0c5942" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:12:59.726137 13310 raft_consensus.cc:3037] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.726263 14223 consensus_queue.cc:237] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:12:59.728876 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33490610e3b4819a61ef4bbfe0c5942" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:12:59.732925 13310 raft_consensus.cc:2454] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:12:59.737681 9949 catalog_manager.cc:5484] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } }
I20240503 02:12:59.779201 10358 consensus_queue.cc:237] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:12:59.782289 10225 raft_consensus.cc:1261] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:12:59.782613 10292 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 206e6a3850ff4242b30fa10c3f751fd2. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:59.782836 10093 raft_consensus.cc:1261] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:59.783202 14223 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:59.784237 14222 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 02:12:59.789350 14347 raft_consensus.cc:482] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:12:59.789583 14347 raft_consensus.cc:504] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.790431 14347 leader_election.cc:288] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:59.790424 14223 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:12:59.791719 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d856c707315b47e996ee604a5ff66b31" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:12:59.792076 13174 raft_consensus.cc:2454] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:59.792642 12843 leader_election.cc:302] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:59.792712 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d856c707315b47e996ee604a5ff66b31" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:12:59.792984 14347 raft_consensus.cc:2781] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:12:59.793105 14347 raft_consensus.cc:482] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:12:59.793087 13040 raft_consensus.cc:2454] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:12:59.793182 14347 raft_consensus.cc:3037] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.798254 14347 raft_consensus.cc:504] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.798761 9949 catalog_manager.cc:5484] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:59.799415 14347 leader_election.cc:288] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:12:59.801031 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d856c707315b47e996ee604a5ff66b31" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:12:59.801290 13174 raft_consensus.cc:3037] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.804677 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d856c707315b47e996ee604a5ff66b31" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:12:59.804970 13040 raft_consensus.cc:3037] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:12:59.807435 13174 raft_consensus.cc:2454] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:59.810083 12843 leader_election.cc:302] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:12:59.810673 14347 raft_consensus.cc:2781] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:12:59.811417 14347 raft_consensus.cc:686] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:12:59.811942 14347 consensus_queue.cc:237] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:12:59.813954 10093 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:12:59.818770 13040 raft_consensus.cc:2454] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:12:59.828127 10225 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:12:59.838156 9952 catalog_manager.cc:5484] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:12:59.889873 10489 consensus_queue.cc:237] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:12:59.890007 9952 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:12:59.893402 10358 raft_consensus.cc:1261] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:12:59.894722 10093 raft_consensus.cc:1261] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:12:59.895020 10427 consensus_peers.cc:485] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a2aac20a18f4444a9ff79642b23c89d9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:59.895680 13487 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:59.896476 13487 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:12:59.899816 13040 raft_consensus.cc:1261] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:59.900573 13310 raft_consensus.cc:1261] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:59.900617 14347 consensus_queue.cc:1035] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.901705 14347 consensus_queue.cc:1035] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:12:59.903141 13487 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:12:59.905365 10358 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:12:59.913637 9952 catalog_manager.cc:5484] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:12:59.920740 10093 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:12:59.933439 9946 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (table rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c]). Expected version 1 got 0
I20240503 02:12:59.937649 9946 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (table rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c]). Expected version 1 got 0
I20240503 02:12:59.938689 14381 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:59.939392 14381 tablet.cc:1620] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:12:59.947788 14383 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:59.949039 14383 tablet.cc:1620] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:12:59.950238 9949 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (table rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c]). Expected version 1 got 0
I20240503 02:12:59.954895 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c] alter complete (version 1)
I20240503 02:12:59.961115 14385 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:12:59.962688 14385 tablet.cc:1620] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:12:59.963908 9946 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (table rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c]). Expected version 1 got 0
I20240503 02:12:59.976541 13039 consensus_queue.cc:237] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:12:59.979633 13310 raft_consensus.cc:1261] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:12:59.979669 12908 raft_consensus.cc:1261] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:12:59.980012 12975 consensus_peers.cc:485] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d9a312f6358e41d7abe66b2a6339e7f4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:12:59.980345 14220 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:59.980783 14390 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:12:59.984220 14219 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:12:59.985060 13310 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:12:59.993842 9952 catalog_manager.cc:5484] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:00.001478 12908 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:00.021729 10094 consensus_queue.cc:237] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:00.024468 10358 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.024605 10489 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:00.025236 10029 consensus_peers.cc:485] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5547b33324db46c886287b4ee5035e43. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.025314 14400 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.026557 14285 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.029645 14104 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:00.030526 10358 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:00.030392 10489 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:00.037132 9946 catalog_manager.cc:5484] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:00.045526 14285 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.048645 14104 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.067126 9949 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:00.075027 10358 consensus_queue.cc:237] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:00.078598 10094 raft_consensus.cc:1261] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:00.079053 10292 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 12da2248f416404da29a4557b90daef7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.081516 14381 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:00.081663 10225 raft_consensus.cc:1261] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.082226 14381 tablet.cc:1620] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:00.082289 14222 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.083094 14362 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.084671 14385 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:00.085326 14385 tablet.cc:1620] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:13:00.092250 14383 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:00.092943 14383 tablet.cc:1620] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:00.097404 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=a974c992d7634430956100e26ff6436c] alter complete (version 2)
I20240503 02:13:00.117376 14362 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:00.121459 10094 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:00.123036 10225 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:00.125640 9952 catalog_manager.cc:5484] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:00.157585 10491 consensus_queue.cc:237] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:00.160432 10358 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:00.160482 12908 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
W20240503 02:13:00.160717 10427 consensus_peers.cc:485] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 36457442a8e24ece9a1c2d8cdab89b78. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.161284 13487 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:00.161711 14377 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:00.165007 14346 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:00.165796 12908 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:00.165789 10358 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:00.172788 9946 catalog_manager.cc:5484] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 7 to 8, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 2 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:00.195099 10358 consensus_queue.cc:237] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:00.198498 10094 raft_consensus.cc:1261] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.199249 14362 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.199471 10225 raft_consensus.cc:1261] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.201045 14359 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.201133 14362 consensus_queue.cc:1035] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.203089 14222 consensus_queue.cc:1035] T b33490610e3b4819a61ef4bbfe0c5942 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.204252 9952 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:00.220490 14359 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:00.222024 10227 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:00.222510 14381 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:00.223565 14383 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:00.230209 9952 catalog_manager.cc:5484] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:00.233469 14385 tablet.cc:1722] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:00.234650 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=a974c992d7634430956100e26ff6436c] alter complete (version 3)
W20240503 02:13:00.251664 10292 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 20366a3e8dcc4996bfdd1d60ef1be077. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.260860 10093 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:00.274088 14438 ts_tablet_manager.cc:902] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:00.300189 14438 tablet_copy_client.cc:308] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:00.301234 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 206e6a3850ff4242b30fa10c3f751fd2 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:39119)
I20240503 02:13:00.301498 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 206e6a3850ff4242b30fa10c3f751fd2 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:39119: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-206e6a3850ff4242b30fa10c3f751fd2
I20240503 02:13:00.304656 10380 tablet_copy_source_session.cc:205] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.309993 14438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 206e6a3850ff4242b30fa10c3f751fd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.311360 10227 consensus_queue.cc:237] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:00.316215 10358 raft_consensus.cc:1261] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.317657 10491 raft_consensus.cc:1261] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.317785 14438 tablet_copy_client.cc:783] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:00.318153 14438 tablet_copy_client.cc:647] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
W20240503 02:13:00.318311 10162 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 731649c884c3461ab562590f012727c5. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.318388 13787 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.319605 13787 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.320581 14438 tablet_copy_client.cc:523] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:00.324008 14334 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:00.324826 10491 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:00.325520 14438 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:00.327541 10358 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:00.335786 9949 catalog_manager.cc:5484] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:00.342763 14347 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.345875 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:00.346303 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:00.352730 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8
I20240503 02:13:00.354454 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:00.355968 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_18"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:00.360955 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:00.362349 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:00 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:00.362829 14456 tablet_replica.cc:330] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:00.363421 14457 tablet_replica.cc:330] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:00.364256 14457 raft_consensus.cc:2229] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.364961 14457 raft_consensus.cc:2258] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.365876 14456 raft_consensus.cc:2229] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:00.367112 14457 ts_tablet_manager.cc:1875] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.367945 14456 raft_consensus.cc:2258] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.372170 14456 ts_tablet_manager.cc:1875] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.373823 13805 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:00.382172 14456 ts_tablet_manager.cc:1888] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:00.382437 14456 log.cc:1196] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/40a8ca5ec12f4e2baa9d5b5f7d0f00e8
I20240503 02:13:00.383176 14456 ts_tablet_manager.cc:1909] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:13:00.384956 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (table rebalancer_dynamic_table_new_name [id=a974c992d7634430956100e26ff6436c]) successfully deleted
I20240503 02:13:00.402132 14457 ts_tablet_manager.cc:1888] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:00.403978 14457 log.cc:1196] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/40a8ca5ec12f4e2baa9d5b5f7d0f00e8
I20240503 02:13:00.404656 14457 ts_tablet_manager.cc:1909] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:00.405148 14454 tablet_replica.cc:330] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:00.405570 14454 raft_consensus.cc:2229] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:00.406026 14454 raft_consensus.cc:2258] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:00.408854 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (table rebalancer_dynamic_table_new_name [id=a974c992d7634430956100e26ff6436c]) successfully deleted
I20240503 02:13:00.412657 10491 consensus_queue.cc:237] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:00.423370 14461 ts_tablet_manager.cc:902] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:00.424896 10072 tablet_service.cc:1467] Processing CreateTablet for tablet ce47128b75a645e8a9f6e4cbc9d8d021 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.425535 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce47128b75a645e8a9f6e4cbc9d8d021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.426919 13152 tablet_service.cc:1467] Processing CreateTablet for tablet ce47128b75a645e8a9f6e4cbc9d8d021 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.427453 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce47128b75a645e8a9f6e4cbc9d8d021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.427451 13020 tablet_service.cc:1467] Processing CreateTablet for tablet e9def231ec4a43349b5786776e6c57b7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.427929 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9def231ec4a43349b5786776e6c57b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.431656 13288 tablet_service.cc:1467] Processing CreateTablet for tablet ce47128b75a645e8a9f6e4cbc9d8d021 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.432209 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce47128b75a645e8a9f6e4cbc9d8d021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.432708 12887 tablet_service.cc:1467] Processing CreateTablet for tablet e9def231ec4a43349b5786776e6c57b7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.433211 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9def231ec4a43349b5786776e6c57b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.434356 14438 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:00.434835 14438 tablet_bootstrap.cc:492] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:00.435168 14438 ts_tablet_manager.cc:1367] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.110s user 0.061s sys 0.009s
I20240503 02:13:00.436241 14438 raft_consensus.cc:348] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:00.436576 14438 raft_consensus.cc:729] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:00.436574 12888 tablet_service.cc:1467] Processing CreateTablet for tablet c2a328e790e844b5aa9a1cf0fa16ee8c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.436854 14438 consensus_queue.cc:260] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:00.437042 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a328e790e844b5aa9a1cf0fa16ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.437398 13154 tablet_service.cc:1467] Processing CreateTablet for tablet e9def231ec4a43349b5786776e6c57b7 (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.437572 14438 ts_tablet_manager.cc:1398] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:00.437891 13154 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9def231ec4a43349b5786776e6c57b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.438457 14463 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:00.440884 13153 tablet_service.cc:1467] Processing CreateTablet for tablet c2a328e790e844b5aa9a1cf0fa16ee8c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.441336 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a328e790e844b5aa9a1cf0fa16ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.441699 14463 tablet_bootstrap.cc:654] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.442734 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-206e6a3850ff4242b30fa10c3f751fd2 received from {username='slave'} at 127.4.5.72:39119
I20240503 02:13:00.442927 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-206e6a3850ff4242b30fa10c3f751fd2 on tablet 206e6a3850ff4242b30fa10c3f751fd2 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:00.431656 13290 tablet_service.cc:1467] Processing CreateTablet for tablet c2a328e790e844b5aa9a1cf0fa16ee8c (DEFAULT_TABLE table=rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:00.447481 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a328e790e844b5aa9a1cf0fa16ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.454275 14454 ts_tablet_manager.cc:1875] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:00.466315 14467 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:00.467711 9951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:00.473861 10358 raft_consensus.cc:1261] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.474097 10227 raft_consensus.cc:1261] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.474963 14377 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.481920 14346 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.489626 13487 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:00.491431 14468 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:00.493052 10225 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:00.500301 14468 tablet_bootstrap.cc:654] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.504873 9949 catalog_manager.cc:5484] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:00.518725 14461 tablet_copy_client.cc:308] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:13:00.520768 14463 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:00.520745 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet a2aac20a18f4444a9ff79642b23c89d9 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:33045)
I20240503 02:13:00.520977 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet a2aac20a18f4444a9ff79642b23c89d9 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:33045: session id = f93b103af083471b8e8c0376e9148aad-a2aac20a18f4444a9ff79642b23c89d9
I20240503 02:13:00.522135 14463 ts_tablet_manager.cc:1367] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.084s user 0.013s sys 0.026s
I20240503 02:13:00.523528 14463 raft_consensus.cc:348] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:00.524036 10511 tablet_copy_source_session.cc:205] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.525406 14463 raft_consensus.cc:374] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.525804 14463 raft_consensus.cc:729] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.526504 14461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a2aac20a18f4444a9ff79642b23c89d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.526805 14463 consensus_queue.cc:260] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:00.530609 14463 ts_tablet_manager.cc:1398] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:00.533816 14465 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:00.535291 13153 tablet_service.cc:1467] Processing CreateTablet for tablet af4eab9fe3ba4c1ba10c082d7c54442f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434]), partition=
I20240503 02:13:00.535732 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4eab9fe3ba4c1ba10c082d7c54442f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.536401 10358 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:00.537245 10072 tablet_service.cc:1467] Processing CreateTablet for tablet af4eab9fe3ba4c1ba10c082d7c54442f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434]), partition=
I20240503 02:13:00.537680 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4eab9fe3ba4c1ba10c082d7c54442f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.550072 14463 tablet_bootstrap.cc:492] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
W20240503 02:13:00.550465 10424 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 73ae9bc7a96149259bd807369c2daacd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.551254 14461 tablet_copy_client.cc:783] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:00.551863 14461 tablet_copy_client.cc:647] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:00.555510 14461 tablet_copy_client.cc:523] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:00.560302 14463 tablet_bootstrap.cc:654] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.560660 14480 ts_tablet_manager.cc:902] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:00.533413 14474 ts_tablet_manager.cc:902] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:00.539629 13020 tablet_service.cc:1467] Processing CreateTablet for tablet af4eab9fe3ba4c1ba10c082d7c54442f (DEFAULT_TABLE table=rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434]), partition=
I20240503 02:13:00.565650 14461 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:00.566228 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af4eab9fe3ba4c1ba10c082d7c54442f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.572014 14438 ts_tablet_manager.cc:902] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:00.581614 14465 tablet_bootstrap.cc:654] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.592660 14468 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:00.592792 14466 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:00.594409 14468 ts_tablet_manager.cc:1367] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.104s user 0.001s sys 0.006s
I20240503 02:13:00.598317 14463 tablet_bootstrap.cc:492] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:00.599129 14463 ts_tablet_manager.cc:1367] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.006s
I20240503 02:13:00.598927 14474 tablet_copy_client.cc:308] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:00.600240 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 5547b33324db46c886287b4ee5035e43 from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:39295)
I20240503 02:13:00.600492 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 5547b33324db46c886287b4ee5035e43 from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:39295: session id = 8d454a3607d042dc9559ed59c3ccd84b-5547b33324db46c886287b4ee5035e43
I20240503 02:13:00.604082 14438 tablet_copy_client.cc:308] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:00.605074 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 12da2248f416404da29a4557b90daef7 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:39119)
I20240503 02:13:00.605304 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 12da2248f416404da29a4557b90daef7 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:39119: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-12da2248f416404da29a4557b90daef7
I20240503 02:13:00.602312 14468 raft_consensus.cc:348] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:00.606648 14468 raft_consensus.cc:374] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.606844 10115 tablet_copy_source_session.cc:205] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.607344 14468 raft_consensus.cc:729] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.608207 14466 tablet_bootstrap.cc:654] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.608628 10380 tablet_copy_source_session.cc:205] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.608837 14474 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5547b33324db46c886287b4ee5035e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.609745 14465 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:00.610023 14465 ts_tablet_manager.cc:1367] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.076s user 0.006s sys 0.000s
I20240503 02:13:00.610589 14438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 12da2248f416404da29a4557b90daef7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.611330 14454 ts_tablet_manager.cc:1888] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:00.611760 14467 tablet_bootstrap.cc:654] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.615042 14465 raft_consensus.cc:348] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.613574 14463 raft_consensus.cc:348] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:00.615916 14463 raft_consensus.cc:374] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.617146 14463 raft_consensus.cc:729] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.617385 14465 raft_consensus.cc:374] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.617786 14463 consensus_queue.cc:260] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:00.618160 14465 raft_consensus.cc:729] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.619432 14468 consensus_queue.cc:260] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:00.624826 14465 consensus_queue.cc:260] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.620040 14438 tablet_copy_client.cc:783] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:00.626642 14454 log.cc:1196] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/40a8ca5ec12f4e2baa9d5b5f7d0f00e8
I20240503 02:13:00.626842 14438 tablet_copy_client.cc:647] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:00.627390 14454 ts_tablet_manager.cc:1909] T 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:13:00.629190 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 40a8ca5ec12f4e2baa9d5b5f7d0f00e8 (table rebalancer_dynamic_table_new_name [id=a974c992d7634430956100e26ff6436c]) successfully deleted
I20240503 02:13:00.630096 14438 tablet_copy_client.cc:523] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:00.631376 14480 tablet_copy_client.cc:308] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:00.631549 14468 ts_tablet_manager.cc:1398] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20240503 02:13:00.633273 14463 ts_tablet_manager.cc:1398] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.034s user 0.001s sys 0.002s
I20240503 02:13:00.635511 14465 ts_tablet_manager.cc:1398] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20240503 02:13:00.636878 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet d9a312f6358e41d7abe66b2a6339e7f4 from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:40827)
I20240503 02:13:00.637391 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet d9a312f6358e41d7abe66b2a6339e7f4 from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:40827: session id = 031a0afb01b6469299fa829fa3467ef7-d9a312f6358e41d7abe66b2a6339e7f4
I20240503 02:13:00.637648 14438 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:00.638216 14465 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:00.645646 13061 tablet_copy_source_session.cc:205] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.647958 14465 tablet_bootstrap.cc:654] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.651212 14466 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:00.651448 14466 ts_tablet_manager.cc:1367] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.003s
I20240503 02:13:00.651500 14480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d9a312f6358e41d7abe66b2a6339e7f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.649152 14468 tablet_bootstrap.cc:492] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:00.657518 14466 raft_consensus.cc:348] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.657871 14466 raft_consensus.cc:374] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.658006 14466 raft_consensus.cc:729] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.658349 14466 consensus_queue.cc:260] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.659837 14466 ts_tablet_manager.cc:1398] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.008s user 0.003s sys 0.001s
I20240503 02:13:00.660216 14466 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:00.665107 14480 tablet_copy_client.cc:783] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:00.665546 14480 tablet_copy_client.cc:647] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:00.673869 14466 tablet_bootstrap.cc:654] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.682142 14480 tablet_copy_client.cc:523] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:00.682248 14468 tablet_bootstrap.cc:654] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.702805 14480 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:00.708083 14466 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:00.708482 14466 ts_tablet_manager.cc:1367] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.048s user 0.004s sys 0.001s
I20240503 02:13:00.709741 14466 raft_consensus.cc:348] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:00.710139 14466 raft_consensus.cc:374] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.711961 14474 tablet_copy_client.cc:783] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:00.712373 14474 tablet_copy_client.cc:647] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:00.723361 14506 ts_tablet_manager.cc:902] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:00.726687 14400 raft_consensus.cc:482] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.726938 14400 raft_consensus.cc:504] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:00.727810 14400 leader_election.cc:288] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:00.732100 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4eab9fe3ba4c1ba10c082d7c54442f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:00.732662 14466 raft_consensus.cc:729] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
W20240503 02:13:00.733152 10029 leader_election.cc:341] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:00.733078 14466 consensus_queue.cc:260] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:00.734983 14507 ts_tablet_manager.cc:902] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:00.735153 14466 ts_tablet_manager.cc:1398] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.026s user 0.000s sys 0.002s
I20240503 02:13:00.738466 10225 consensus_queue.cc:237] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:00.729307 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4eab9fe3ba4c1ba10c082d7c54442f" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:00.800426 14465 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:00.800437 10489 raft_consensus.cc:1261] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:00.800555 10029 leader_election.cc:341] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:00.800726 10029 leader_election.cc:302] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7; no voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:00.800776 14465 ts_tablet_manager.cc:1367] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.163s user 0.005s sys 0.004s
I20240503 02:13:00.801146 10359 raft_consensus.cc:1261] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:00.801290 14400 raft_consensus.cc:2726] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:00.801708 14465 raft_consensus.cc:348] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:00.801990 14465 raft_consensus.cc:374] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.802063 14465 raft_consensus.cc:729] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.801908 14334 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.802326 14465 consensus_queue.cc:260] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:00.802984 14465 ts_tablet_manager.cc:1398] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:00.735460 14474 tablet_copy_client.cc:523] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:13:00.805946 10162 consensus_peers.cc:485] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0dc2733b737b431684ce439c3d3fa762. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:00.806324 14334 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:00.807744 14474 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:00.795585 14468 tablet_bootstrap.cc:492] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:00.819429 14468 ts_tablet_manager.cc:1367] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.183s user 0.005s sys 0.001s
I20240503 02:13:00.827965 14467 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:00.828226 14467 ts_tablet_manager.cc:1367] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.362s user 0.005s sys 0.011s
I20240503 02:13:00.829586 14467 raft_consensus.cc:348] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.829999 14467 raft_consensus.cc:374] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.830166 14467 raft_consensus.cc:729] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.830586 14467 consensus_queue.cc:260] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.828663 14468 raft_consensus.cc:348] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:00.831009 14468 raft_consensus.cc:374] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:00.831159 14468 raft_consensus.cc:729] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:00.831516 14468 consensus_queue.cc:260] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:00.833673 14468 ts_tablet_manager.cc:1398] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:13:00.840387 14467 ts_tablet_manager.cc:1398] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.012s user 0.000s sys 0.003s
I20240503 02:13:00.844123 14467 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:00.845413 14480 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:00.850034 14480 tablet_bootstrap.cc:492] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:00.850612 14480 ts_tablet_manager.cc:1367] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.148s user 0.023s sys 0.002s
I20240503 02:13:00.852324 14197 raft_consensus.cc:482] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.852663 14197 raft_consensus.cc:504] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.853617 14197 leader_election.cc:288] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:00.854099 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a328e790e844b5aa9a1cf0fa16ee8c" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:00.854413 12905 raft_consensus.cc:2454] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:00.854511 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a328e790e844b5aa9a1cf0fa16ee8c" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:00.854760 13172 raft_consensus.cc:2454] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:00.854955 13243 leader_election.cc:302] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:00.856384 14467 tablet_bootstrap.cc:654] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:00.857389 14197 raft_consensus.cc:2781] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.857532 14197 raft_consensus.cc:482] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.857650 14197 raft_consensus.cc:3037] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.859267 13309 raft_consensus.cc:1203] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:00.861297 14197 raft_consensus.cc:504] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.867094 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a328e790e844b5aa9a1cf0fa16ee8c" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:00.867554 12905 raft_consensus.cc:3037] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.870220 14506 tablet_copy_client.cc:308] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:00.871707 12905 raft_consensus.cc:2454] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:00.872570 13243 leader_election.cc:302] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:00.874173 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c2a328e790e844b5aa9a1cf0fa16ee8c" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:00.874460 13172 raft_consensus.cc:3037] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.878346 13172 raft_consensus.cc:2454] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:00.879144 14197 leader_election.cc:288] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:00.879391 14514 ts_tablet_manager.cc:902] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:00.879922 14197 raft_consensus.cc:2781] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.880165 14197 raft_consensus.cc:686] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:00.880748 14197 consensus_queue.cc:237] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:00.883131 14480 raft_consensus.cc:348] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:00.884356 14480 raft_consensus.cc:729] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:00.885540 14480 consensus_queue.cc:260] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:00.887534 14334 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:00.889039 14480 ts_tablet_manager.cc:1398] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:13:00.891999 10491 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:00.893447 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-d9a312f6358e41d7abe66b2a6339e7f4 received from {username='slave'} at 127.4.5.65:40827
I20240503 02:13:00.893672 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-d9a312f6358e41d7abe66b2a6339e7f4 on tablet d9a312f6358e41d7abe66b2a6339e7f4 with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:00.904227 10358 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:00.916404 14479 raft_consensus.cc:482] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.916674 14479 raft_consensus.cc:504] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:00.918061 14479 leader_election.cc:288] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:00.918157 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4eab9fe3ba4c1ba10c082d7c54442f" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:00.918704 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4eab9fe3ba4c1ba10c082d7c54442f" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
W20240503 02:13:00.918830 12976 leader_election.cc:341] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:00.918985 10094 raft_consensus.cc:2454] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:00.919494 12975 leader_election.cc:302] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b; no voters: f93b103af083471b8e8c0376e9148aad
I20240503 02:13:00.920598 14479 raft_consensus.cc:2781] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.920737 14479 raft_consensus.cc:482] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.920843 14479 raft_consensus.cc:3037] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.926653 14479 raft_consensus.cc:504] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:00.927841 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4eab9fe3ba4c1ba10c082d7c54442f" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:00.928272 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af4eab9fe3ba4c1ba10c082d7c54442f" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
W20240503 02:13:00.928386 12976 leader_election.cc:341] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:00.928611 10094 raft_consensus.cc:3037] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.931819 14479 leader_election.cc:288] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:00.933480 10094 raft_consensus.cc:2454] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:00.934190 12975 leader_election.cc:302] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b; no voters: f93b103af083471b8e8c0376e9148aad
I20240503 02:13:00.934767 14390 raft_consensus.cc:2781] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.943341 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 731649c884c3461ab562590f012727c5 from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:38111)
I20240503 02:13:00.943724 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 731649c884c3461ab562590f012727c5 from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:38111: session id = 8d454a3607d042dc9559ed59c3ccd84b-731649c884c3461ab562590f012727c5
I20240503 02:13:00.948984 9949 catalog_manager.cc:5484] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:00.949381 10248 tablet_copy_source_session.cc:205] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.951838 14506 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 731649c884c3461ab562590f012727c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.960647 9949 catalog_manager.cc:5484] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:00.962384 14479 raft_consensus.cc:482] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.962602 14479 raft_consensus.cc:504] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:00.964536 14479 leader_election.cc:288] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:00.965199 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9def231ec4a43349b5786776e6c57b7" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:00.965620 14507 tablet_copy_client.cc:308] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:13:00.965863 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9def231ec4a43349b5786776e6c57b7" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
W20240503 02:13:00.965979 12976 leader_election.cc:341] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:00.966168 12908 raft_consensus.cc:2454] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:00.966928 12973 leader_election.cc:302] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: f93b103af083471b8e8c0376e9148aad
I20240503 02:13:00.967561 14479 raft_consensus.cc:2781] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:00.966450 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet 36457442a8e24ece9a1c2d8cdab89b78 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:33045)
I20240503 02:13:00.967859 14479 raft_consensus.cc:482] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:00.967981 14479 raft_consensus.cc:3037] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.967995 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet 36457442a8e24ece9a1c2d8cdab89b78 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:33045: session id = f93b103af083471b8e8c0376e9148aad-36457442a8e24ece9a1c2d8cdab89b78
I20240503 02:13:00.962687 14506 tablet_copy_client.cc:783] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:00.971132 14506 tablet_copy_client.cc:647] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:00.972139 10511 tablet_copy_source_session.cc:205] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:00.975267 14507 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 36457442a8e24ece9a1c2d8cdab89b78. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:00.976503 14506 tablet_copy_client.cc:523] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:00.990079 14479 raft_consensus.cc:504] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:00.993077 14530 raft_consensus.cc:482] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:00.993291 14530 raft_consensus.cc:504] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:00.993865 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9def231ec4a43349b5786776e6c57b7" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:00.994203 12908 raft_consensus.cc:3037] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:00.994362 14530 leader_election.cc:288] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:00.994949 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce47128b75a645e8a9f6e4cbc9d8d021" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
W20240503 02:13:00.997387 10029 leader_election.cc:341] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:00.997615 12908 raft_consensus.cc:2454] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:00.998178 12973 leader_election.cc:302] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:00.998600 14477 raft_consensus.cc:2781] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:00.998801 14477 raft_consensus.cc:686] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:00.999218 14477 consensus_queue.cc:237] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:00.999389 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9def231ec4a43349b5786776e6c57b7" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
W20240503 02:13:01.000090 12976 leader_election.cc:341] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.000001 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce47128b75a645e8a9f6e4cbc9d8d021" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:01.000303 14479 leader_election.cc:288] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:01.000519 13309 raft_consensus.cc:2454] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:00.992264 14506 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:01.001194 10026 leader_election.cc:302] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: f93b103af083471b8e8c0376e9148aad
I20240503 02:13:01.001833 14530 raft_consensus.cc:2781] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.001974 14530 raft_consensus.cc:482] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.002076 14530 raft_consensus.cc:3037] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.006372 14530 raft_consensus.cc:504] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:01.009312 13173 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce47128b75a645e8a9f6e4cbc9d8d021" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
W20240503 02:13:01.010071 10029 leader_election.cc:341] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.010102 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce47128b75a645e8a9f6e4cbc9d8d021" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:01.010382 13310 raft_consensus.cc:3037] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.014539 14467 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:01.014830 14467 ts_tablet_manager.cc:1367] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.171s user 0.010s sys 0.013s
I20240503 02:13:01.016110 14467 raft_consensus.cc:348] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:01.017843 13310 raft_consensus.cc:2454] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:01.018425 10026 leader_election.cc:302] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: f93b103af083471b8e8c0376e9148aad
I20240503 02:13:01.018846 14285 raft_consensus.cc:2781] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.019009 14285 raft_consensus.cc:686] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:13:01.019034 14467 raft_consensus.cc:374] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.019133 14467 raft_consensus.cc:729] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.019373 14285 consensus_queue.cc:237] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:01.019436 14467 consensus_queue.cc:260] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:01.023118 14390 raft_consensus.cc:686] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:01.023224 14530 leader_election.cc:288] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:01.023593 14390 consensus_queue.cc:237] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:01.027900 14467 ts_tablet_manager.cc:1398] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20240503 02:13:01.028568 14507 tablet_copy_client.cc:783] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.035017 14467 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:01.038256 14467 tablet_bootstrap.cc:654] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.043283 9949 catalog_manager.cc:5484] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.036109 14507 tablet_copy_client.cc:647] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.045755 14474 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.046429 14474 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:01.046921 14474 ts_tablet_manager.cc:1367] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.239s user 0.047s sys 0.008s
I20240503 02:13:01.048019 14474 raft_consensus.cc:348] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:01.049428 14507 tablet_copy_client.cc:523] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.050782 14474 raft_consensus.cc:729] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:01.051151 14474 consensus_queue.cc:260] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:01.051337 10094 raft_consensus.cc:1203] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:01.052145 14474 ts_tablet_manager.cc:1398] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:01.053066 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-5547b33324db46c886287b4ee5035e43 received from {username='slave'} at 127.4.5.70:39295
I20240503 02:13:01.053282 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-5547b33324db46c886287b4ee5035e43 on tablet 5547b33324db46c886287b4ee5035e43 with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:01.081651 14507 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:01.089609 9952 catalog_manager.cc:5484] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.090507 12908 consensus_queue.cc:237] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:01.090006 14545 ts_tablet_manager.cc:902] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:01.099139 9952 catalog_manager.cc:5484] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:01.101390 13173 raft_consensus.cc:1261] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.101670 14514 tablet_copy_client.cc:308] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:01.103199 13805 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:01.108915 10489 raft_consensus.cc:1261] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.110219 14467 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:01.110224 14347 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:01.110450 14467 ts_tablet_manager.cc:1367] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.006s
I20240503 02:13:01.111615 14467 raft_consensus.cc:348] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:01.111949 14467 raft_consensus.cc:374] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.112133 14467 raft_consensus.cc:729] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.112552 14467 consensus_queue.cc:260] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:01.113396 14467 ts_tablet_manager.cc:1398] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:01.113787 14467 tablet_bootstrap.cc:492] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
W20240503 02:13:01.120915 12842 consensus_peers.cc:485] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 559ae0e1e196444f9d50420d0f799754. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:01.123077 14467 tablet_bootstrap.cc:654] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.124625 14347 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:01.127647 13173 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:01.140087 14467 tablet_bootstrap.cc:492] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:01.140364 14467 ts_tablet_manager.cc:1367] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.004s
I20240503 02:13:01.141427 14467 raft_consensus.cc:348] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:01.143033 14545 tablet_copy_client.cc:308] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:13:01.146307 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet 73ae9bc7a96149259bd807369c2daacd from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:41551)
I20240503 02:13:01.146545 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet 73ae9bc7a96149259bd807369c2daacd from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:41551: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-73ae9bc7a96149259bd807369c2daacd
I20240503 02:13:01.149718 14467 raft_consensus.cc:374] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.149888 14467 raft_consensus.cc:729] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.149919 10511 tablet_copy_source_session.cc:205] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.152292 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 73ae9bc7a96149259bd807369c2daacd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.152694 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 20366a3e8dcc4996bfdd1d60ef1be077 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:56263)
I20240503 02:13:01.152947 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 20366a3e8dcc4996bfdd1d60ef1be077 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:56263: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-20366a3e8dcc4996bfdd1d60ef1be077
I20240503 02:13:01.154752 9951 catalog_manager.cc:5484] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.159817 14545 tablet_copy_client.cc:783] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.161121 10380 tablet_copy_source_session.cc:205] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.163028 14514 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 20366a3e8dcc4996bfdd1d60ef1be077. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.171407 14514 tablet_copy_client.cc:783] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.171888 14514 tablet_copy_client.cc:647] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.174069 14545 tablet_copy_client.cc:647] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.180262 14545 tablet_copy_client.cc:523] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.181891 14506 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.182536 14506 tablet_bootstrap.cc:492] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:01.182982 14506 ts_tablet_manager.cc:1367] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.191s user 0.051s sys 0.008s
I20240503 02:13:01.183187 14514 tablet_copy_client.cc:523] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.184073 14506 raft_consensus.cc:348] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:01.184523 14506 raft_consensus.cc:729] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:01.184799 14506 consensus_queue.cc:260] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:01.189218 14545 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:01.189512 14514 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:01.193042 14506 ts_tablet_manager.cc:1398] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:01.196110 14467 consensus_queue.cc:260] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:01.197516 10248 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-731649c884c3461ab562590f012727c5 received from {username='slave'} at 127.4.5.70:38111
I20240503 02:13:01.197707 10248 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-731649c884c3461ab562590f012727c5 on tablet 731649c884c3461ab562590f012727c5 with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:01.201278 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:01.202481 14467 ts_tablet_manager.cc:1398] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.062s user 0.003s sys 0.000s
I20240503 02:13:01.224323 10491 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:01.228121 9946 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet af4eab9fe3ba4c1ba10c082d7c54442f (table rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434]). Expected version 1 got 0
I20240503 02:13:01.241254 14196 raft_consensus.cc:482] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.241530 14196 raft_consensus.cc:504] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:01.242436 14196 leader_election.cc:288] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:01.242940 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9def231ec4a43349b5786776e6c57b7" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:01.243120 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e9def231ec4a43349b5786776e6c57b7" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:01.243299 12908 raft_consensus.cc:2379] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f93b103af083471b8e8c0376e9148aad in current term 1: Already voted for candidate 8d454a3607d042dc9559ed59c3ccd84b in this term.
I20240503 02:13:01.243888 13110 leader_election.cc:302] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:01.244678 14196 raft_consensus.cc:3037] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.250698 14196 raft_consensus.cc:2726] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:01.257952 14564 ts_tablet_manager.cc:902] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:01.270195 10094 raft_consensus.cc:1261] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.271183 14533 consensus_queue.cc:1035] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.271445 13173 raft_consensus.cc:3037] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.276635 13173 raft_consensus.cc:1261] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.277397 14533 consensus_queue.cc:1035] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.284868 14521 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.299638 9946 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet af4eab9fe3ba4c1ba10c082d7c54442f (table rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434]). Expected version 1 got 0
I20240503 02:13:01.310994 14578 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:01.311740 14578 tablet.cc:1620] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:01.316401 14438 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.316994 14438 tablet_bootstrap.cc:492] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:01.319797 14438 ts_tablet_manager.cc:1367] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.682s user 0.069s sys 0.010s
I20240503 02:13:01.320645 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434] alter complete (version 1)
I20240503 02:13:01.324266 9949 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet af4eab9fe3ba4c1ba10c082d7c54442f (table rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434]). Expected version 1 got 0
I20240503 02:13:01.321666 14438 raft_consensus.cc:348] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:01.327548 13309 consensus_queue.cc:237] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:01.327803 14438 raft_consensus.cc:729] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:01.328758 14438 consensus_queue.cc:260] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:01.330058 14438 ts_tablet_manager.cc:1398] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20240503 02:13:01.332975 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-12da2248f416404da29a4557b90daef7 received from {username='slave'} at 127.4.5.72:39119
I20240503 02:13:01.333182 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-12da2248f416404da29a4557b90daef7 on tablet 12da2248f416404da29a4557b90daef7 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:01.346589 14581 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:01.347489 14581 tablet.cc:1620] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:01.366181 12908 raft_consensus.cc:1261] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.367676 13173 raft_consensus.cc:1261] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.368657 14197 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:01.371937 13039 raft_consensus.cc:1203] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.373713 14197 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:01.388790 14477 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.390161 14545 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.391816 14545 tablet_bootstrap.cc:492] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:01.392386 14545 ts_tablet_manager.cc:1367] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.203s user 0.051s sys 0.010s
I20240503 02:13:01.393738 14545 raft_consensus.cc:348] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:01.394266 14545 raft_consensus.cc:729] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:01.394625 14545 consensus_queue.cc:260] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:01.397876 14545 ts_tablet_manager.cc:1398] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:01.398674 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-73ae9bc7a96149259bd807369c2daacd received from {username='slave'} at 127.4.5.72:41551
I20240503 02:13:01.398847 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-73ae9bc7a96149259bd807369c2daacd on tablet 73ae9bc7a96149259bd807369c2daacd with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:01.401758 14562 raft_consensus.cc:482] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.402380 14562 raft_consensus.cc:504] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:01.403385 14579 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:01.404049 14579 tablet.cc:1620] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:01.408484 9952 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet af4eab9fe3ba4c1ba10c082d7c54442f (table rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434]). Expected version 1 got 0
I20240503 02:13:01.409070 14197 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.416157 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce47128b75a645e8a9f6e4cbc9d8d021" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:01.416869 13309 raft_consensus.cc:2379] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f93b103af083471b8e8c0376e9148aad in current term 1: Already voted for candidate 031a0afb01b6469299fa829fa3467ef7 in this term.
I20240503 02:13:01.416734 10093 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ce47128b75a645e8a9f6e4cbc9d8d021" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:01.417510 14562 leader_election.cc:288] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:01.417804 13109 leader_election.cc:302] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:01.428455 14562 raft_consensus.cc:3037] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.428936 14533 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.428905 14197 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:01.432931 14562 raft_consensus.cc:2726] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:01.437880 12907 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:01.438203 9946 catalog_manager.cc:5484] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:01.443348 9949 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:01.445298 14564 tablet_copy_client.cc:308] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:01.446909 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 0dc2733b737b431684ce439c3d3fa762 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:44307)
I20240503 02:13:01.446801 14400 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.447142 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 0dc2733b737b431684ce439c3d3fa762 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:44307: session id = f93b103af083471b8e8c0376e9148aad-0dc2733b737b431684ce439c3d3fa762
I20240503 02:13:01.454612 10248 tablet_copy_source_session.cc:205] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:01.455386 14578 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:01.455999 14578 tablet.cc:1620] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:01.458712 14564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0dc2733b737b431684ce439c3d3fa762. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.466343 14579 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:01.467773 14579 tablet.cc:1620] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:01.469097 14223 raft_consensus.cc:1050] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:01.469910 14581 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:01.470494 14581 tablet.cc:1620] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:01.470242 14223 consensus_queue.cc:237] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:01.471514 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=fd5a70dfa54e4991a6ee34e55b6a5434] alter complete (version 2)
I20240503 02:13:01.491992 10093 raft_consensus.cc:1261] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:01.493541 13039 raft_consensus.cc:1203] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.501668 14359 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:01.507313 10225 raft_consensus.cc:1261] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:01.509063 14359 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:01.523856 13310 raft_consensus.cc:1203] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.524626 14564 tablet_copy_client.cc:783] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:01.525418 14564 tablet_copy_client.cc:647] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:01.525910 13310 raft_consensus.cc:1261] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:01.527278 14390 raft_consensus.cc:1050] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
W20240503 02:13:01.527604 13245 consensus_peers.cc:485] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e6bced8dfb104a5eb0db971cd11aad79. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:01.522230 13173 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:01.528502 14390 consensus_queue.cc:237] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:01.530125 14222 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:01.535065 14514 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.535638 14514 tablet_bootstrap.cc:492] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:01.535610 14530 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:01.535993 14514 ts_tablet_manager.cc:1367] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.347s user 0.053s sys 0.008s
I20240503 02:13:01.537027 14514 raft_consensus.cc:348] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:01.549603 14514 raft_consensus.cc:729] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:01.550019 14514 consensus_queue.cc:260] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:01.552152 10093 raft_consensus.cc:1261] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:01.552798 14223 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.554343 14514 ts_tablet_manager.cc:1398] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20240503 02:13:01.555052 14564 tablet_copy_client.cc:523] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:01.555238 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-20366a3e8dcc4996bfdd1d60ef1be077 received from {username='slave'} at 127.4.5.69:56263
I20240503 02:13:01.555387 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-20366a3e8dcc4996bfdd1d60ef1be077 on tablet 20366a3e8dcc4996bfdd1d60ef1be077 with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:01.557737 14477 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:01.559000 13310 raft_consensus.cc:1261] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:01.558952 10093 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.559000 10225 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.565762 9952 catalog_manager.cc:5484] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.573403 14461 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.579178 14461 tablet_bootstrap.cc:492] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:01.579813 14461 ts_tablet_manager.cc:1367] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 1.019s user 0.063s sys 0.015s
I20240503 02:13:01.583917 14479 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:01.584728 13309 raft_consensus.cc:1203] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.584807 14461 raft_consensus.cc:348] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:01.586619 14461 raft_consensus.cc:729] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:01.586980 14461 consensus_queue.cc:260] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:01.589884 14461 ts_tablet_manager.cc:1398] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:01.592264 12908 raft_consensus.cc:1261] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:01.593302 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-a2aac20a18f4444a9ff79642b23c89d9 received from {username='slave'} at 127.4.5.71:33045
I20240503 02:13:01.593549 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session f93b103af083471b8e8c0376e9148aad-a2aac20a18f4444a9ff79642b23c89d9 on tablet a2aac20a18f4444a9ff79642b23c89d9 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:01.595093 14390 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:01.615381 13310 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.620070 10359 consensus_queue.cc:237] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:01.622977 10092 raft_consensus.cc:1261] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:01.623826 14222 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:01.627074 14222 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.632186 10092 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.634183 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 206e6a3850ff4242b30fa10c3f751fd2 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:01.635777 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:01.642138 9951 catalog_manager.cc:5484] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.644608 13310 raft_consensus.cc:1261] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:01.645395 14222 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:01.649701 10206 tablet_service.cc:1514] Processing DeleteTablet for tablet 206e6a3850ff4242b30fa10c3f751fd2 with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:01.649533 13310 raft_consensus.cc:2932] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:01.659019 14390 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.661366 14612 tablet_replica.cc:330] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:01.661648 14612 raft_consensus.cc:2229] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:01.670342 14612 raft_consensus.cc:2258] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:01.673345 14564 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:01.674324 14612 ts_tablet_manager.cc:1875] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:01.675119 14611 ts_tablet_manager.cc:902] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:01.683393 14578 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:01.689152 14579 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:01.692214 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:01.697449 12908 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.700969 13308 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.704919 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 254af82e74a045f8a9c9d9de48cdee3e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.704984 12888 tablet_service.cc:1467] Processing CreateTablet for tablet 254af82e74a045f8a9c9d9de48cdee3e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.705528 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 254af82e74a045f8a9c9d9de48cdee3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.705552 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 254af82e74a045f8a9c9d9de48cdee3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.711210 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 53a6d97e7b054cecb8b29f282d08349b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.711812 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a6d97e7b054cecb8b29f282d08349b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.715531 14581 tablet.cc:1722] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:01.715732 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 254af82e74a045f8a9c9d9de48cdee3e (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.716291 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 254af82e74a045f8a9c9d9de48cdee3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.717052 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbc1a14c000465e9bb642fcc1e84611 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.717568 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbc1a14c000465e9bb642fcc1e84611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.719437 14507 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:01.720010 14507 tablet_bootstrap.cc:492] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:01.720336 14507 ts_tablet_manager.cc:1367] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.639s user 0.067s sys 0.003s
I20240503 02:13:01.715777 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbc1a14c000465e9bb642fcc1e84611 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.720919 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 53a6d97e7b054cecb8b29f282d08349b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.721237 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbc1a14c000465e9bb642fcc1e84611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.721341 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a6d97e7b054cecb8b29f282d08349b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.722126 14507 raft_consensus.cc:348] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:01.722441 14507 raft_consensus.cc:729] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:01.722656 14507 consensus_queue.cc:260] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:01.723510 14507 ts_tablet_manager.cc:1398] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:01.725340 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-36457442a8e24ece9a1c2d8cdab89b78 received from {username='slave'} at 127.4.5.71:33045
I20240503 02:13:01.725528 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session f93b103af083471b8e8c0376e9148aad-36457442a8e24ece9a1c2d8cdab89b78 on tablet 36457442a8e24ece9a1c2d8cdab89b78 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:01.734437 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 9fbc1a14c000465e9bb642fcc1e84611 (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.735098 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbc1a14c000465e9bb642fcc1e84611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.746639 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 53a6d97e7b054cecb8b29f282d08349b (DEFAULT_TABLE table=rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:01.747201 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 53a6d97e7b054cecb8b29f282d08349b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:01.755177 9944 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=fd5a70dfa54e4991a6ee34e55b6a5434] alter complete (version 3)
I20240503 02:13:01.756800 10092 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.764269 9946 catalog_manager.cc:5484] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.801703 14618 ts_tablet_manager.cc:902] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:01.803278 14617 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:01.807832 13039 consensus_queue.cc:237] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:01.810995 10093 raft_consensus.cc:1261] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:01.811880 14477 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:01.814172 14616 tablet_bootstrap.cc:492] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:01.815455 13308 raft_consensus.cc:1261] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:01.816171 10092 consensus_queue.cc:237] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:01.816421 14390 consensus_queue.cc:1035] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:01.816995 14620 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:01.817107 14390 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.817925 10094 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.819356 14620 tablet_bootstrap.cc:654] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.819842 14616 tablet_bootstrap.cc:654] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.820045 14621 tablet_bootstrap.cc:492] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:01.822547 14621 tablet_bootstrap.cc:654] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.828311 14617 tablet_bootstrap.cc:654] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.830322 13308 raft_consensus.cc:2932] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.831346 14612 ts_tablet_manager.cc:1888] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:01.833258 14612 log.cc:1196] T 206e6a3850ff4242b30fa10c3f751fd2 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/206e6a3850ff4242b30fa10c3f751fd2
I20240503 02:13:01.835657 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 206e6a3850ff4242b30fa10c3f751fd2 (table rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]) successfully deleted
I20240503 02:13:01.841753 9946 catalog_manager.cc:5484] T d9a312f6358e41d7abe66b2a6339e7f4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:01.849534 14617 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:01.849692 14620 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:01.850240 14620 ts_tablet_manager.cc:1367] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.003s
I20240503 02:13:01.850883 14621 tablet_bootstrap.cc:492] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:01.851539 14620 raft_consensus.cc:348] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:01.851821 14620 raft_consensus.cc:374] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.851914 14620 raft_consensus.cc:729] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.851740 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet d9a312f6358e41d7abe66b2a6339e7f4 with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:01.852183 14620 consensus_queue.cc:260] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:01.855105 14617 ts_tablet_manager.cc:1367] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.000s
I20240503 02:13:01.857779 14616 tablet_bootstrap.cc:492] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:01.858031 14616 ts_tablet_manager.cc:1367] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20240503 02:13:01.860133 14621 ts_tablet_manager.cc:1367] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.000s
I20240503 02:13:01.863057 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d9a312f6358e41d7abe66b2a6339e7f4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:01.863516 14615 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:01.864070 14616 raft_consensus.cc:348] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:01.864387 14616 raft_consensus.cc:374] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.864537 14616 raft_consensus.cc:729] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.864893 14616 consensus_queue.cc:260] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:01.865124 14631 tablet_replica.cc:330] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:01.866509 14616 ts_tablet_manager.cc:1398] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:01.866964 14616 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:01.867092 14617 raft_consensus.cc:348] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:01.869158 14616 tablet_bootstrap.cc:654] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.871233 14621 raft_consensus.cc:348] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:01.871702 14617 raft_consensus.cc:374] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.872270 14617 raft_consensus.cc:729] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.873961 13172 raft_consensus.cc:1203] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.874647 14617 consensus_queue.cc:260] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:01.875180 14621 raft_consensus.cc:374] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.875746 14621 raft_consensus.cc:729] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.876554 14621 consensus_queue.cc:260] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:01.877858 14617 ts_tablet_manager.cc:1398] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:13:01.881458 14285 raft_consensus.cc:1050] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:01.882609 14285 consensus_queue.cc:237] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:01.883489 14621 ts_tablet_manager.cc:1398] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.023s user 0.000s sys 0.003s
I20240503 02:13:01.884652 14617 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:01.887400 14615 tablet_bootstrap.cc:654] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.887969 13039 raft_consensus.cc:1261] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:01.890295 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9abc528ca83640c3bf66022cf2c9b914. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:01.890434 9944 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:01.890756 9944 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:01.891992 14400 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:01.898252 13308 raft_consensus.cc:1261] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:01.899936 14400 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:01.908744 14617 tablet_bootstrap.cc:654] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.913199 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet af4eab9fe3ba4c1ba10c082d7c54442f
I20240503 02:13:01.931797 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet af4eab9fe3ba4c1ba10c082d7c54442f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:01 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:01.931847 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet af4eab9fe3ba4c1ba10c082d7c54442f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:01 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:01.932427 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet af4eab9fe3ba4c1ba10c082d7c54442f with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:01 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:01.933281 14223 raft_consensus.cc:482] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.933539 14223 raft_consensus.cc:504] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:01.935184 14223 leader_election.cc:288] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:01.936666 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254af82e74a045f8a9c9d9de48cdee3e" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
W20240503 02:13:01.937739 10295 leader_election.cc:341] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.938747 14615 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:01.941105 14615 ts_tablet_manager.cc:1367] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.078s user 0.005s sys 0.003s
I20240503 02:13:01.942292 14223 raft_consensus.cc:1050] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:01.943378 14223 consensus_queue.cc:237] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:01.943818 14617 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:01.944037 14617 ts_tablet_manager.cc:1367] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.059s user 0.006s sys 0.000s
I20240503 02:13:01.945737 14617 raft_consensus.cc:348] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:01.949504 14615 raft_consensus.cc:348] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:01.949967 14615 raft_consensus.cc:374] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.950136 14615 raft_consensus.cc:729] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.950703 14615 consensus_queue.cc:260] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:01.954823 14400 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:01.956496 13310 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:01.959507 14617 raft_consensus.cc:374] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:01.959620 14617 raft_consensus.cc:729] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:01.961473 14593 raft_consensus.cc:482] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.961592 14640 tablet_replica.cc:330] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:01.961714 14593 raft_consensus.cc:504] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:01.962810 14615 ts_tablet_manager.cc:1398] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.021s user 0.002s sys 0.001s
I20240503 02:13:01.962805 14593 leader_election.cc:288] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:01.963675 14615 tablet_bootstrap.cc:492] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:01.963980 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a6d97e7b054cecb8b29f282d08349b" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:01.965795 14615 tablet_bootstrap.cc:654] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:01.960680 14617 consensus_queue.cc:260] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:01.958703 14641 tablet_replica.cc:330] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:01.967979 14641 raft_consensus.cc:2229] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:01.968559 14641 raft_consensus.cc:2258] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:01.969666 14641 ts_tablet_manager.cc:1875] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:13:01.974035 13246 leader_election.cc:341] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.976007 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a6d97e7b054cecb8b29f282d08349b" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:01.976290 10489 raft_consensus.cc:2454] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:01.976428 14617 ts_tablet_manager.cc:1398] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20240503 02:13:01.976884 9951 catalog_manager.cc:5484] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:01.961889 14640 raft_consensus.cc:2229] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:01.977743 14640 raft_consensus.cc:2258] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:01.979323 13243 leader_election.cc:302] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters: 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:01.980018 14593 raft_consensus.cc:2781] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:01.980162 14593 raft_consensus.cc:482] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:01.980245 14593 raft_consensus.cc:3037] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.981490 14640 ts_tablet_manager.cc:1875] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:01.983294 14593 raft_consensus.cc:504] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:01.984707 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a6d97e7b054cecb8b29f282d08349b" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
W20240503 02:13:01.985471 13246 leader_election.cc:341] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:01.987258 14593 leader_election.cc:288] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:01.988106 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a6d97e7b054cecb8b29f282d08349b" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:01.988365 10489 raft_consensus.cc:3037] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:01.992113 10489 raft_consensus.cc:2454] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:01.992892 13243 leader_election.cc:302] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters: 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:01.993870 14521 raft_consensus.cc:2781] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:01.994033 14521 raft_consensus.cc:686] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:01.994331 14521 consensus_queue.cc:237] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:01.997493 14222 raft_consensus.cc:482] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:01.997746 14222 raft_consensus.cc:504] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:01.998272 14611 tablet_copy_client.cc:308] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:01.998626 14222 leader_election.cc:288] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:01.999522 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbc1a14c000465e9bb642fcc1e84611" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:01.999738 14642 tablet_replica.cc:330] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:02.000046 14642 raft_consensus.cc:2229] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:13:02.000484 10292 leader_election.cc:341] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.000768 14642 raft_consensus.cc:2258] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.004956 14615 tablet_bootstrap.cc:492] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:02.006814 13040 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.007354 10489 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.007369 10359 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.007791 14530 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:01.873142 14620 ts_tablet_manager.cc:1398] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.023s user 0.001s sys 0.001s
I20240503 02:13:01.912807 12905 raft_consensus.cc:1203] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:01.927567 13805 raft_consensus.cc:482] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.009714 14631 raft_consensus.cc:2229] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.010030 14631 raft_consensus.cc:2258] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.010111 14620 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:02.009740 13805 raft_consensus.cc:504] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:02.013276 14618 tablet_copy_client.cc:308] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.72:33671
I20240503 02:13:02.013651 13805 leader_election.cc:288] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:02.016709 14615 ts_tablet_manager.cc:1367] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.001s
I20240503 02:13:02.017824 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254af82e74a045f8a9c9d9de48cdee3e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:13:02.017747 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254af82e74a045f8a9c9d9de48cdee3e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:02.018100 10359 raft_consensus.cc:2454] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:02.018426 13038 raft_consensus.cc:2454] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:02.018426 13330 tablet_copy_service.cc:140] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Received BeginTabletCopySession request for tablet e6bced8dfb104a5eb0db971cd11aad79 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:56309)
I20240503 02:13:02.018654 12842 leader_election.cc:302] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:02.018712 13330 tablet_copy_service.cc:161] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Beginning new tablet copy session on tablet e6bced8dfb104a5eb0db971cd11aad79 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:56309: session id = eff8e8d11a4447218741bb994aa41082-e6bced8dfb104a5eb0db971cd11aad79
I20240503 02:13:02.019974 13805 raft_consensus.cc:2781] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.020112 13805 raft_consensus.cc:482] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.020196 13805 raft_consensus.cc:3037] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.023239 14641 ts_tablet_manager.cc:1888] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:02.023427 14641 log.cc:1196] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/af4eab9fe3ba4c1ba10c082d7c54442f
I20240503 02:13:02.023415 13805 raft_consensus.cc:504] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:02.023924 14641 ts_tablet_manager.cc:1909] T af4eab9fe3ba4c1ba10c082d7c54442f P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:02.024051 13330 tablet_copy_source_session.cc:205] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:02.025004 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254af82e74a045f8a9c9d9de48cdee3e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:02.025249 13038 raft_consensus.cc:3037] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.026986 14620 tablet_bootstrap.cc:654] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.009245 10092 raft_consensus.cc:1261] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.028168 14618 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e6bced8dfb104a5eb0db971cd11aad79. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.028353 13038 raft_consensus.cc:2454] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:02.029830 12843 leader_election.cc:302] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:02.011744 14285 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.021s
I20240503 02:13:02.033414 14223 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.033846 14285 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.017764 14615 raft_consensus.cc:348] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:02.035620 14615 raft_consensus.cc:374] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.035976 14615 raft_consensus.cc:729] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.036669 14615 consensus_queue.cc:260] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:02.039170 14615 ts_tablet_manager.cc:1398] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20240503 02:13:02.040413 14346 raft_consensus.cc:1050] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:02.041448 14346 consensus_queue.cc:237] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:02.042229 13039 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:01.942895 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254af82e74a045f8a9c9d9de48cdee3e" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:02.047863 14640 ts_tablet_manager.cc:1888] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:02.047240 12908 raft_consensus.cc:2379] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 47f901e617024baaa1865d12261ef84f in current term 1: Already voted for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in this term.
I20240503 02:13:02.039741 14631 ts_tablet_manager.cc:1875] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:02.049790 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet af4eab9fe3ba4c1ba10c082d7c54442f (table rebalancer_dynamic_table_new_name [id=fd5a70dfa54e4991a6ee34e55b6a5434]) successfully deleted
I20240503 02:13:02.051084 14640 log.cc:1196] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/af4eab9fe3ba4c1ba10c082d7c54442f
I20240503 02:13:02.054040 10227 raft_consensus.cc:1261] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.054778 14222 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.055546 13787 raft_consensus.cc:1050] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:02.056838 13787 consensus_queue.cc:237] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:02.057219 13310 raft_consensus.cc:1261] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.058044 14377 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.059815 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet 559ae0e1e196444f9d50420d0f799754 from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:58973)
I20240503 02:13:02.060065 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet 559ae0e1e196444f9d50420d0f799754 from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:58973: session id = 031a0afb01b6469299fa829fa3467ef7-559ae0e1e196444f9d50420d0f799754
I20240503 02:13:02.061460 10357 raft_consensus.cc:1261] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.061913 10225 raft_consensus.cc:1261] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.062494 14377 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.063151 14377 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.063472 12928 tablet_copy_source_session.cc:205] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:02.068084 14644 raft_consensus.cc:2781] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.069768 14640 ts_tablet_manager.cc:1909] T af4eab9fe3ba4c1ba10c082d7c54442f P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:13:02.070675 10292 leader_election.cc:302] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47f901e617024baaa1865d12261ef84f; no voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:02.071149 14222 raft_consensus.cc:3037] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.071388 14611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 559ae0e1e196444f9d50420d0f799754. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.072957 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet af4eab9fe3ba4c1ba10c082d7c54442f (table rebalancer_dynamic_table_new_name [id=fd5a70dfa54e4991a6ee34e55b6a5434]) successfully deleted
I20240503 02:13:02.074604 14222 raft_consensus.cc:2726] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:02.078020 14618 tablet_copy_client.cc:783] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:02.078418 14618 tablet_copy_client.cc:647] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:02.081666 14618 tablet_copy_client.cc:523] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:02.068686 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbc1a14c000465e9bb642fcc1e84611" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:02.084933 14644 raft_consensus.cc:686] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:02.086548 14618 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:02.087347 14644 consensus_queue.cc:237] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
W20240503 02:13:02.088513 10292 leader_election.cc:341] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.088740 10292 leader_election.cc:302] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47f901e617024baaa1865d12261ef84f; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:02.089349 14222 raft_consensus.cc:2726] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:02.093709 13805 leader_election.cc:288] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:02.093835 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "254af82e74a045f8a9c9d9de48cdee3e" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:02.093959 13038 raft_consensus.cc:1261] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.097497 10358 raft_consensus.cc:2454] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:02.099671 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:02.101310 13487 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.106676 13172 raft_consensus.cc:1203] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad [term 2 LEARNER]: Deduplicated request from leader. Original: 2.7->[2.8-2.8] Dedup: 2.8->[]
I20240503 02:13:02.111675 14359 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.118175 10357 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.122004 10092 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.131457 9951 catalog_manager.cc:5484] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.132671 13308 raft_consensus.cc:1261] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.139396 14631 ts_tablet_manager.cc:1888] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:02.139837 14631 log.cc:1196] T d9a312f6358e41d7abe66b2a6339e7f4 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/d9a312f6358e41d7abe66b2a6339e7f4
I20240503 02:13:02.142879 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet d9a312f6358e41d7abe66b2a6339e7f4 (table rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]) successfully deleted
I20240503 02:13:02.144771 10227 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.155218 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 1623a9f120c347bd9dd7ee138e9b10ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]), partition=
I20240503 02:13:02.155853 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1623a9f120c347bd9dd7ee138e9b10ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.158360 12888 tablet_service.cc:1467] Processing CreateTablet for tablet 1623a9f120c347bd9dd7ee138e9b10ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]), partition=
I20240503 02:13:02.158881 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1623a9f120c347bd9dd7ee138e9b10ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.171623 14334 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.077s
I20240503 02:13:02.144678 9949 catalog_manager.cc:5484] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.175498 10225 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.187502 14530 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.192725 14611 tablet_copy_client.cc:783] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:02.193549 10489 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.196259 13308 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.196832 14611 tablet_copy_client.cc:647] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:02.201586 13039 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.202860 14222 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.206552 9951 catalog_manager.cc:5484] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.207228 14642 ts_tablet_manager.cc:1875] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:02.209702 10359 raft_consensus.cc:1261] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.210424 9943 catalog_manager.cc:5484] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.213347 14616 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:02.213577 14616 ts_tablet_manager.cc:1367] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.347s user 0.011s sys 0.008s
I20240503 02:13:02.218695 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 1623a9f120c347bd9dd7ee138e9b10ff (DEFAULT_TABLE table=rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]), partition=
I20240503 02:13:02.219219 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1623a9f120c347bd9dd7ee138e9b10ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.221724 14618 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:02.222517 14618 tablet_bootstrap.cc:492] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:02.222965 14674 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.227084 14618 ts_tablet_manager.cc:1367] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.141s user 0.015s sys 0.000s
I20240503 02:13:02.229560 10489 consensus_queue.cc:237] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:02.232235 14618 raft_consensus.cc:348] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:02.232744 14618 raft_consensus.cc:729] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:02.233232 13310 raft_consensus.cc:1261] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.234370 14377 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.233117 14618 consensus_queue.cc:260] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:02.236950 10357 raft_consensus.cc:1261] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.237710 10490 raft_consensus.cc:1261] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.238170 13487 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.239081 10357 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.239885 14618 ts_tablet_manager.cc:1398] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:13:02.244755 10094 consensus_queue.cc:237] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:02.237680 14669 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.249740 13039 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.250535 14285 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.252182 14620 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:02.253244 13330 tablet_copy_service.cc:342] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-e6bced8dfb104a5eb0db971cd11aad79 received from {username='slave'} at 127.4.5.66:56309
I20240503 02:13:02.253487 13330 tablet_copy_service.cc:433] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: ending tablet copy session eff8e8d11a4447218741bb994aa41082-e6bced8dfb104a5eb0db971cd11aad79 on tablet e6bced8dfb104a5eb0db971cd11aad79 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:02.254611 14611 tablet_copy_client.cc:523] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:02.255898 10358 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.256603 10225 raft_consensus.cc:1203] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:02.259347 14611 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:02.260848 14674 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.264250 10359 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.267496 14400 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.273710 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5547b33324db46c886287b4ee5035e43 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:02.278537 14530 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.280200 13038 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
W20240503 02:13:02.282233 12843 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.5.70:43707, user_credentials={real_user=slave}} blocked reactor thread for 220657us
I20240503 02:13:02.284819 14620 ts_tablet_manager.cc:1367] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.242s user 0.004s sys 0.008s
I20240503 02:13:02.285593 13039 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.286352 14620 raft_consensus.cc:348] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:02.286705 14620 raft_consensus.cc:374] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.286798 14620 raft_consensus.cc:729] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.287127 14620 consensus_queue.cc:260] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:02.287829 14620 ts_tablet_manager.cc:1398] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:02.287786 9944 catalog_manager.cc:5484] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:02.288561 14620 tablet_bootstrap.cc:492] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:02.290009 13309 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.290952 14377 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.291886 14611 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:02.292466 14611 tablet_bootstrap.cc:492] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:02.292805 14611 ts_tablet_manager.cc:1367] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.003s
I20240503 02:13:02.294225 14611 raft_consensus.cc:348] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:02.295954 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 73ae9bc7a96149259bd807369c2daacd with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:02.301362 14611 raft_consensus.cc:729] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:02.301898 14611 consensus_queue.cc:260] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:02.303831 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5547b33324db46c886287b4ee5035e43 with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:02.304008 14611 ts_tablet_manager.cc:1398] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:13:02.321259 14685 tablet_replica.cc:330] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:02.324396 10359 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.327152 9946 catalog_manager.cc:5484] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.330277 9951 catalog_manager.cc:5484] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:02.333887 10359 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.334270 14685 raft_consensus.cc:2229] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.334604 14685 raft_consensus.cc:2258] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.336102 9952 catalog_manager.cc:5484] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:02.337386 14685 ts_tablet_manager.cc:1875] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:02.345070 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 73ae9bc7a96149259bd807369c2daacd with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:02.345237 10225 consensus_queue.cc:237] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:02.350351 13039 raft_consensus.cc:1261] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.351260 14669 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.354368 14669 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.355895 14680 tablet_bootstrap.cc:492] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:02.325199 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-559ae0e1e196444f9d50420d0f799754 received from {username='slave'} at 127.4.5.65:58973
I20240503 02:13:02.356057 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-559ae0e1e196444f9d50420d0f799754 on tablet 559ae0e1e196444f9d50420d0f799754 with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:02.358436 13038 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.359831 14667 raft_consensus.cc:1050] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:02.360913 14667 consensus_queue.cc:237] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:02.372280 14685 ts_tablet_manager.cc:1888] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:02.372638 14685 log.cc:1196] T 5547b33324db46c886287b4ee5035e43 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/5547b33324db46c886287b4ee5035e43
I20240503 02:13:02.375211 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 731649c884c3461ab562590f012727c5 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:02.375882 14620 tablet_bootstrap.cc:654] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.375777 13310 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.376765 10491 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.380017 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 5547b33324db46c886287b4ee5035e43 (table rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]) successfully deleted
I20240503 02:13:02.380844 10359 raft_consensus.cc:1261] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.381737 14688 ts_tablet_manager.cc:902] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:02.385053 14669 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.388418 10359 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:02.389430 13172 raft_consensus.cc:1261] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.390220 14377 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.403062 14686 tablet_replica.cc:330] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:02.405169 14686 raft_consensus.cc:2229] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.405378 10359 raft_consensus.cc:1261] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.405534 10092 raft_consensus.cc:1261] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.405745 14686 raft_consensus.cc:2258] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.406402 14667 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.407039 9946 catalog_manager.cc:5484] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:02.411568 14667 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.413915 14620 tablet_bootstrap.cc:492] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:02.414655 14620 ts_tablet_manager.cc:1367] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.126s user 0.005s sys 0.000s
I20240503 02:13:02.418771 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 731649c884c3461ab562590f012727c5 with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:02.419940 14620 raft_consensus.cc:348] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.421411 14620 raft_consensus.cc:374] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.422433 14620 raft_consensus.cc:729] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.423130 14620 consensus_queue.cc:260] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.428865 14620 ts_tablet_manager.cc:1398] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.014s user 0.002s sys 0.001s
I20240503 02:13:02.431591 14222 raft_consensus.cc:1050] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:02.431790 14688 tablet_copy_client.cc:308] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:02.434082 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 9abc528ca83640c3bf66022cf2c9b914 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:60077)
I20240503 02:13:02.433808 14222 consensus_queue.cc:237] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:02.434324 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 9abc528ca83640c3bf66022cf2c9b914 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:60077: session id = 47f901e617024baaa1865d12261ef84f-9abc528ca83640c3bf66022cf2c9b914
I20240503 02:13:02.438350 10115 tablet_copy_source_session.cc:205] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:02.443058 14688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9abc528ca83640c3bf66022cf2c9b914. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:02.444640 10227 raft_consensus.cc:1261] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.445701 14668 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.445695 14667 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.450476 10092 raft_consensus.cc:1261] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.452841 14644 raft_consensus.cc:482] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.451910 14222 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.452965 10094 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.456180 14644 raft_consensus.cc:504] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:02.458242 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbc1a14c000465e9bb642fcc1e84611" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:13:02.458554 10359 raft_consensus.cc:2454] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:02.459096 12842 leader_election.cc:302] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:02.459733 14651 raft_consensus.cc:2781] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.459913 14651 raft_consensus.cc:482] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.460076 14651 raft_consensus.cc:3037] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.453576 14688 tablet_copy_client.cc:783] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:02.462225 9946 catalog_manager.cc:5484] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.463071 14688 tablet_copy_client.cc:647] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:02.463374 14651 raft_consensus.cc:504] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:02.463995 14390 raft_consensus.cc:482] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.464197 14390 raft_consensus.cc:504] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:02.465204 14390 leader_election.cc:288] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:02.466539 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a6d97e7b054cecb8b29f282d08349b" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:02.472854 10359 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.478652 14688 tablet_copy_client.cc:523] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:02.479063 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbc1a14c000465e9bb642fcc1e84611" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:02.479624 14644 leader_election.cc:288] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:02.482941 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbc1a14c000465e9bb642fcc1e84611" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:02.483371 10359 raft_consensus.cc:3037] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.485417 12907 raft_consensus.cc:1261] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:02.488126 13309 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fbc1a14c000465e9bb642fcc1e84611" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:02.489122 14651 leader_election.cc:288] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:02.494057 10359 raft_consensus.cc:2454] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:02.494748 14222 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:02.496047 12842 leader_election.cc:302] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:02.503692 10359 consensus_queue.cc:237] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:02.506390 14688 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:02.521804 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "53a6d97e7b054cecb8b29f282d08349b" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:02.522225 10489 raft_consensus.cc:2379] T 53a6d97e7b054cecb8b29f282d08349b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d454a3607d042dc9559ed59c3ccd84b in current term 1: Already voted for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in this term.
I20240503 02:13:02.386335 14616 raft_consensus.cc:348] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:02.523052 14616 raft_consensus.cc:374] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.523245 14616 raft_consensus.cc:729] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.523468 14651 raft_consensus.cc:2781] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.523721 14616 consensus_queue.cc:260] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:02.524264 13308 raft_consensus.cc:2454] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:02.525589 12973 leader_election.cc:302] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850
I20240503 02:13:02.526022 14390 raft_consensus.cc:3037] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.529779 14616 ts_tablet_manager.cc:1398] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.316s user 0.003s sys 0.000s
I20240503 02:13:02.529392 14668 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:02.530088 14390 raft_consensus.cc:2726] T 53a6d97e7b054cecb8b29f282d08349b P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:02.533313 10094 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:02.540591 14651 raft_consensus.cc:686] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:02.544116 14593 consensus_queue.cc:1035] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.546015 14685 tablet_replica.cc:330] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:02.546288 14685 raft_consensus.cc:2229] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.546576 14685 raft_consensus.cc:2258] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.548095 14685 ts_tablet_manager.cc:1875] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:02.548147 14680 tablet_bootstrap.cc:654] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.541024 14651 consensus_queue.cc:237] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:02.549208 9943 catalog_manager.cc:5484] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:02.555410 10292 consensus_peers.cc:485] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35aea508f59f4348b05d93a5afdde4ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:02.555452 14642 ts_tablet_manager.cc:1888] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:02.555610 14642 log.cc:1196] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/af4eab9fe3ba4c1ba10c082d7c54442f
I20240503 02:13:02.556121 14642 ts_tablet_manager.cc:1909] T af4eab9fe3ba4c1ba10c082d7c54442f P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:13:02.556495 10225 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:02.557387 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet af4eab9fe3ba4c1ba10c082d7c54442f (table rebalancer_dynamic_table_new_name [id=fd5a70dfa54e4991a6ee34e55b6a5434]) successfully deleted
I20240503 02:13:02.558607 14655 raft_consensus.cc:482] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.558799 14655 raft_consensus.cc:504] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.559695 14655 leader_election.cc:288] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:02.560201 10225 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1623a9f120c347bd9dd7ee138e9b10ff" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
W20240503 02:13:02.562897 12842 leader_election.cc:341] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.559345 14223 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.570920 10491 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:02.576021 14685 ts_tablet_manager.cc:1888] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:02.580686 14685 log.cc:1196] T 731649c884c3461ab562590f012727c5 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/731649c884c3461ab562590f012727c5
I20240503 02:13:02.578881 10227 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:02.582032 14593 consensus_queue.cc:1035] T 53a6d97e7b054cecb8b29f282d08349b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.583416 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 731649c884c3461ab562590f012727c5 (table rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]) successfully deleted
I20240503 02:13:02.587179 14223 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:02.595196 10359 consensus_queue.cc:237] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:02.594957 12907 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:02.601501 12907 raft_consensus.cc:1261] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.602209 14222 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.605149 14359 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:02.608246 10092 raft_consensus.cc:1203] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:02.611084 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 20366a3e8dcc4996bfdd1d60ef1be077 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:02.614598 12907 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:02.614938 9943 catalog_manager.cc:5484] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.624502 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 20366a3e8dcc4996bfdd1d60ef1be077 with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:02.625272 14642 tablet_replica.cc:330] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:02.625574 14642 raft_consensus.cc:2229] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.625898 14642 raft_consensus.cc:2258] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.627446 14642 ts_tablet_manager.cc:1875] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:02.631417 14688 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:02.632115 14688 tablet_bootstrap.cc:492] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:02.633112 14688 ts_tablet_manager.cc:1367] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.126s user 0.020s sys 0.012s
I20240503 02:13:02.634562 14688 raft_consensus.cc:348] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:02.634879 14688 raft_consensus.cc:729] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:02.635326 14688 consensus_queue.cc:260] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:02.636265 14688 ts_tablet_manager.cc:1398] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20240503 02:13:02.636606 12843 leader_election.cc:341] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Not found: Tablet not found: 1623a9f120c347bd9dd7ee138e9b10ff
I20240503 02:13:02.637351 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-9abc528ca83640c3bf66022cf2c9b914 received from {username='slave'} at 127.4.5.67:60077
I20240503 02:13:02.637502 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session 47f901e617024baaa1865d12261ef84f-9abc528ca83640c3bf66022cf2c9b914 on tablet 9abc528ca83640c3bf66022cf2c9b914 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:02.637518 14377 raft_consensus.cc:1050] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:02.638628 14377 consensus_queue.cc:237] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:02.640908 14359 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:02.641920 12843 leader_election.cc:302] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: eff8e8d11a4447218741bb994aa41082, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:02.643632 13805 raft_consensus.cc:2726] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:02.647634 14642 ts_tablet_manager.cc:1888] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:02.647910 14642 log.cc:1196] T 20366a3e8dcc4996bfdd1d60ef1be077 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/20366a3e8dcc4996bfdd1d60ef1be077
I20240503 02:13:02.652825 14680 tablet_bootstrap.cc:492] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:02.652951 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 20366a3e8dcc4996bfdd1d60ef1be077 (table rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]) successfully deleted
I20240503 02:13:02.653204 14680 ts_tablet_manager.cc:1367] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.298s user 0.016s sys 0.010s
I20240503 02:13:02.654454 14680 raft_consensus.cc:348] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.655123 14680 raft_consensus.cc:374] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.655318 14680 raft_consensus.cc:729] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.656133 14680 consensus_queue.cc:260] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.656994 14680 ts_tablet_manager.cc:1398] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:02.659976 13172 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.661938 9943 catalog_manager.cc:5484] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.665182 14686 ts_tablet_manager.cc:1875] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:02.667871 14651 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.676741 12907 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:02.677600 14684 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:02.679353 10491 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:02.680246 10224 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:02.685535 10359 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:02.685875 13174 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad [term 2 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:02.687141 14684 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:02.688784 14651 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.691006 14684 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:13:02.692826 13246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.70:43707, user_credentials={real_user=slave}} blocked reactor thread for 72717.4us
I20240503 02:13:02.705317 13787 raft_consensus.cc:482] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:02.705619 13787 raft_consensus.cc:504] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.706614 13787 leader_election.cc:288] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:02.706912 9943 catalog_manager.cc:5484] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:02.710062 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1623a9f120c347bd9dd7ee138e9b10ff" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:02.710424 12905 raft_consensus.cc:2454] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:02.712419 10159 leader_election.cc:302] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:02.713547 13787 raft_consensus.cc:2781] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:02.713702 13787 raft_consensus.cc:482] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:02.713822 13787 raft_consensus.cc:3037] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.717983 13787 raft_consensus.cc:504] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.720501 10226 raft_consensus.cc:1261] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.721740 14359 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.724788 10224 raft_consensus.cc:2932] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:02.735106 14593 raft_consensus.cc:1050] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:02.736322 14593 consensus_queue.cc:237] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:02.740248 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1623a9f120c347bd9dd7ee138e9b10ff" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:02.740566 12905 raft_consensus.cc:3037] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.743649 12905 raft_consensus.cc:2454] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:02.744644 10159 leader_election.cc:302] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:02.745158 14674 raft_consensus.cc:2781] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:02.745323 14674 raft_consensus.cc:686] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:02.745679 14674 consensus_queue.cc:237] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.746631 10227 raft_consensus.cc:1261] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:02.747443 12905 raft_consensus.cc:1261] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:02.748518 14521 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:02.751252 14720 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:02.755487 13787 leader_election.cc:288] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:02.774710 14594 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:02.775507 10224 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:02.775071 9946 catalog_manager.cc:5484] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.781046 12905 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:02.790364 14686 ts_tablet_manager.cc:1888] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:02.790660 14686 log.cc:1196] T 73ae9bc7a96149259bd807369c2daacd P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/73ae9bc7a96149259bd807369c2daacd
I20240503 02:13:02.790289 10491 consensus_queue.cc:237] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:02.793190 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 73ae9bc7a96149259bd807369c2daacd (table rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]) successfully deleted
I20240503 02:13:02.794428 9946 catalog_manager.cc:5484] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:02.796092 14684 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.803231 10359 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.803094 12905 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.813604 9946 catalog_manager.cc:5484] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 8 to 9, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.814616 10092 raft_consensus.cc:1261] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.817615 14684 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.821278 13171 raft_consensus.cc:1261] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:02.823233 14346 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.831389 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1623a9f120c347bd9dd7ee138e9b10ff" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:02.831665 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet a2aac20a18f4444a9ff79642b23c89d9 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 02:13:02.832733 10162 leader_election.cc:341] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.832739 9944 catalog_manager.cc:5484] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:02.833678 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1623a9f120c347bd9dd7ee138e9b10ff" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:02.834528 10358 consensus_queue.cc:237] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:02.834779 14720 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:13:02.835146 10162 leader_election.cc:341] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:02.839259 14712 tablet_bootstrap.cc:492] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:02.839720 10092 raft_consensus.cc:1261] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.840543 14668 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.842200 14712 tablet_bootstrap.cc:654] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:02.843997 10094 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.847496 14668 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.848831 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet a2aac20a18f4444a9ff79642b23c89d9 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:02.849143 13310 raft_consensus.cc:1261] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.853010 14222 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.854728 13174 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 8 to 9, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.856787 10094 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.858238 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 12da2248f416404da29a4557b90daef7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:02.861548 13310 raft_consensus.cc:2932] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.867190 13166 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:02.883401 10357 raft_consensus.cc:1203] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:02.886039 14712 tablet_bootstrap.cc:492] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:02.882457 10491 consensus_queue.cc:237] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:02.891472 14712 ts_tablet_manager.cc:1367] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.052s user 0.009s sys 0.000s
I20240503 02:13:02.892612 14712 raft_consensus.cc:348] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.893358 9952 catalog_manager.cc:5484] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:02.897990 14712 raft_consensus.cc:374] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:02.898424 14712 raft_consensus.cc:729] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:02.899169 14712 consensus_queue.cc:260] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:02.905164 9952 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:02.906677 12905 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:02.907471 13487 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:02.911818 14667 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.917186 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 36457442a8e24ece9a1c2d8cdab89b78 with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:02.919827 14736 tablet_replica.cc:330] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:02.920248 14736 raft_consensus.cc:2229] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.919806 12907 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.920863 14736 raft_consensus.cc:2258] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.922492 14736 ts_tablet_manager.cc:1875] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:13:02.927484 12976 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 99573.6us
I20240503 02:13:02.937429 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 12da2248f416404da29a4557b90daef7 with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:02.938247 14686 tablet_replica.cc:330] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:02.940840 14686 raft_consensus.cc:2229] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.941200 14686 raft_consensus.cc:2258] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.942771 14686 ts_tablet_manager.cc:1875] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:02.944576 14736 ts_tablet_manager.cc:1888] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:02.945117 14736 log.cc:1196] T a2aac20a18f4444a9ff79642b23c89d9 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/a2aac20a18f4444a9ff79642b23c89d9
I20240503 02:13:02.946251 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet a2aac20a18f4444a9ff79642b23c89d9 (table rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]) successfully deleted
I20240503 02:13:02.953971 14686 ts_tablet_manager.cc:1888] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:02.955058 14686 log.cc:1196] T 12da2248f416404da29a4557b90daef7 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/12da2248f416404da29a4557b90daef7
I20240503 02:13:02.957769 13310 raft_consensus.cc:3037] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.956583 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 12da2248f416404da29a4557b90daef7 (table rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]) successfully deleted
I20240503 02:13:02.935307 9943 catalog_manager.cc:5484] T 36457442a8e24ece9a1c2d8cdab89b78 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 9 to 10, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 2 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.964547 14644 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.965327 14712 ts_tablet_manager.cc:1398] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:13:02.965835 13168 raft_consensus.cc:1261] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:02.967599 13487 consensus_queue.cc:1035] T a2aac20a18f4444a9ff79642b23c89d9 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:02.969298 13166 raft_consensus.cc:3037] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:02.969565 13174 raft_consensus.cc:2932] T a2aac20a18f4444a9ff79642b23c89d9 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:02.974458 12905 raft_consensus.cc:1261] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:02.975153 13787 consensus_queue.cc:1035] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.976547 9949 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]). Expected version 1 got 0
I20240503 02:13:02.985548 10359 consensus_queue.cc:237] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:02.987313 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 36457442a8e24ece9a1c2d8cdab89b78 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:02.987820 14736 tablet_replica.cc:330] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:02.988099 14736 raft_consensus.cc:2229] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:02.988366 14736 raft_consensus.cc:2258] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:02.989972 13166 raft_consensus.cc:1261] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:02.991186 14674 consensus_queue.cc:1035] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:02.992729 14736 ts_tablet_manager.cc:1875] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:03.003001 9944 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]). Expected version 1 got 0
I20240503 02:13:03.006142 10094 raft_consensus.cc:1261] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:03.007400 10295 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: da288f76df0a409eb01ee431d3b83b49. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.008257 14668 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:03.009505 13805 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:03.012661 13172 raft_consensus.cc:1261] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:03.014560 13487 consensus_queue.cc:1035] T 36457442a8e24ece9a1c2d8cdab89b78 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:03.017248 10224 raft_consensus.cc:1261] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:03.018208 14668 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:03.018370 13174 raft_consensus.cc:2932] T 36457442a8e24ece9a1c2d8cdab89b78 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:03.037945 14668 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:03.041927 10227 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:03.046979 9943 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]). Expected version 1 got 0
I20240503 02:13:03.052582 14739 mvcc.cc:205] Tried to move back new op lower bound from 7023420960629501952 to 7023420959729770496. Current Snapshot: MvccSnapshot[applied={T|T < 7023420960629501952}]
I20240503 02:13:03.053465 9943 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]). Expected version 1 got 0
I20240503 02:13:03.054508 9944 catalog_manager.cc:5484] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:03.064916 10094 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:03.068941 14736 ts_tablet_manager.cc:1888] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:03.069113 14736 log.cc:1196] T 36457442a8e24ece9a1c2d8cdab89b78 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/36457442a8e24ece9a1c2d8cdab89b78
I20240503 02:13:03.070446 14750 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:03.070796 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 36457442a8e24ece9a1c2d8cdab89b78 (table rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]) successfully deleted
I20240503 02:13:03.074043 9943 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]). Expected version 1 got 0
I20240503 02:13:03.076534 14748 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:03.081099 14750 tablet.cc:1620] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:03.036397 14564 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.083225 14564 tablet_bootstrap.cc:492] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:03.085379 14748 tablet.cc:1620] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:03.085836 14564 ts_tablet_manager.cc:1367] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 1.413s user 0.063s sys 0.008s
I20240503 02:13:03.086894 14564 raft_consensus.cc:348] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:03.087419 14564 raft_consensus.cc:729] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:03.087757 14564 consensus_queue.cc:260] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:03.102609 9944 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]). Expected version 1 got 0
I20240503 02:13:03.111773 9949 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd] alter complete (version 1)
I20240503 02:13:03.124585 14564 ts_tablet_manager.cc:1398] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20240503 02:13:03.125933 10248 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-0dc2733b737b431684ce439c3d3fa762 received from {username='slave'} at 127.4.5.71:44307
I20240503 02:13:03.126133 10248 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session f93b103af083471b8e8c0376e9148aad-0dc2733b737b431684ce439c3d3fa762 on tablet 0dc2733b737b431684ce439c3d3fa762 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:03.126046 14751 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:03.126868 14751 tablet.cc:1620] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:03.130290 9949 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd]). Expected version 1 got 0
I20240503 02:13:03.151149 14758 ts_tablet_manager.cc:902] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:03.160776 10094 consensus_queue.cc:237] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
W20240503 02:13:03.173116 10026 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e94a48cad1d648b18d709b46566d810b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:03.174505 14758 tablet_copy_client.cc:308] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:03.174821 10357 raft_consensus.cc:1261] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:03.175321 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 35aea508f59f4348b05d93a5afdde4ba from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:39119)
I20240503 02:13:03.175516 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 35aea508f59f4348b05d93a5afdde4ba from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:39119: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-35aea508f59f4348b05d93a5afdde4ba
I20240503 02:13:03.181021 14285 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:03.186551 10380 tablet_copy_source_session.cc:205] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:03.188786 14758 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35aea508f59f4348b05d93a5afdde4ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.188414 14285 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:03.190625 10227 raft_consensus.cc:1261] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:03.193367 14530 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:03.193073 10357 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:03.206748 9943 catalog_manager.cc:5484] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:03.207962 14758 tablet_copy_client.cc:783] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:03.208321 14758 tablet_copy_client.cc:647] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:03.214149 10227 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:03.216552 14758 tablet_copy_client.cc:523] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:03.216595 14651 raft_consensus.cc:1050] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:03.217619 14651 consensus_queue.cc:237] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:03.226336 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:03.226922 14758 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:03.232487 10094 raft_consensus.cc:1261] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:03.241354 14748 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:03.242251 14748 tablet.cc:1620] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:03.242576 14750 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:03.243232 14750 tablet.cc:1620] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:03.246062 13805 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:03.253917 13172 raft_consensus.cc:1203] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:03.256603 14751 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:03.257488 14751 tablet.cc:1620] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:03.295526 9944 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=cdcde9df35a2420bb80ee4a785c59fdd] alter complete (version 2)
I20240503 02:13:03.312505 10491 raft_consensus.cc:1261] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:03.313872 13805 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:03.322538 13166 raft_consensus.cc:1261] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:03.323828 14655 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:03.337451 14530 raft_consensus.cc:1050] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:03.338965 14530 consensus_queue.cc:237] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:03.362409 10359 raft_consensus.cc:1261] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:03.363513 14285 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:03.373687 14758 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:03.374389 14758 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:03.374822 14758 ts_tablet_manager.cc:1367] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.148s user 0.074s sys 0.029s
I20240503 02:13:03.375762 14758 raft_consensus.cc:348] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:03.376073 14758 raft_consensus.cc:729] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:03.376397 14758 consensus_queue.cc:260] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:03.377254 14758 ts_tablet_manager.cc:1398] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:03.381799 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-35aea508f59f4348b05d93a5afdde4ba received from {username='slave'} at 127.4.5.72:39119
I20240503 02:13:03.382055 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-35aea508f59f4348b05d93a5afdde4ba on tablet 35aea508f59f4348b05d93a5afdde4ba with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:03.383339 9944 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:03.388067 13310 raft_consensus.cc:1261] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:03.389595 14285 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:03.411016 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:03.441751 14748 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:03.442721 14751 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:03.473212 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=cdcde9df35a2420bb80ee4a785c59fdd] alter complete (version 3)
I20240503 02:13:03.494505 14750 tablet.cc:1722] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:03.496621 13288 tablet_service.cc:1467] Processing CreateTablet for tablet de58a8fe28a74c28a85c78ea8372ee0a (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.497354 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de58a8fe28a74c28a85c78ea8372ee0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.497325 12888 tablet_service.cc:1467] Processing CreateTablet for tablet de58a8fe28a74c28a85c78ea8372ee0a (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.497828 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de58a8fe28a74c28a85c78ea8372ee0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.506556 13020 tablet_service.cc:1467] Processing CreateTablet for tablet de58a8fe28a74c28a85c78ea8372ee0a (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.507297 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de58a8fe28a74c28a85c78ea8372ee0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.519511 14778 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:03.522701 14778 tablet_bootstrap.cc:654] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.542366 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6056d9c9a949a6907510bd619825f6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.543087 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6056d9c9a949a6907510bd619825f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.555505 13038 raft_consensus.cc:1261] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:03.556545 14781 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:03.556722 14285 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:03.561458 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6056d9c9a949a6907510bd619825f6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.562191 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6056d9c9a949a6907510bd619825f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.568738 14781 tablet_bootstrap.cc:654] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.577417 14564 ts_tablet_manager.cc:902] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:03.578372 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 0f6056d9c9a949a6907510bd619825f6 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.578802 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 09e0373a72654a1082464fac8d428483 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.579023 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6056d9c9a949a6907510bd619825f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.579277 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e0373a72654a1082464fac8d428483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.581732 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 09e0373a72654a1082464fac8d428483 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.582336 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e0373a72654a1082464fac8d428483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.585886 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 09e0373a72654a1082464fac8d428483 (DEFAULT_TABLE table=rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:03.586412 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e0373a72654a1082464fac8d428483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.595499 9952 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:03.595871 9952 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:03.619877 9952 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 1623a9f120c347bd9dd7ee138e9b10ff
I20240503 02:13:03.621419 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 1623a9f120c347bd9dd7ee138e9b10ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:03 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:03.632995 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 1623a9f120c347bd9dd7ee138e9b10ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:03 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:03.634299 14787 tablet_replica.cc:330] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:03.635259 13152 tablet_service.cc:1514] Processing DeleteTablet for tablet 1623a9f120c347bd9dd7ee138e9b10ff with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:03 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:03.639240 14787 raft_consensus.cc:2229] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:03.641022 14787 raft_consensus.cc:2258] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:03.644285 14788 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:03.635090 14790 tablet_replica.cc:330] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:03.645249 14790 raft_consensus.cc:2229] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:03.645623 14790 raft_consensus.cc:2258] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:03.649020 14786 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:03.650396 14787 ts_tablet_manager.cc:1875] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:03.651331 14786 tablet_bootstrap.cc:654] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.653860 14790 ts_tablet_manager.cc:1875] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:03.654785 14737 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:03.657509 13039 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:03.660288 14791 tablet_replica.cc:330] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:03.660598 14791 raft_consensus.cc:2229] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:03.661003 14791 raft_consensus.cc:2258] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:03.664865 14791 ts_tablet_manager.cc:1875] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:03.673260 10359 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:03.680740 14790 ts_tablet_manager.cc:1888] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:03.683036 14669 raft_consensus.cc:1050] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:03.684319 14669 consensus_queue.cc:237] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:03.687711 9949 catalog_manager.cc:5484] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:03.697335 14790 log.cc:1196] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/1623a9f120c347bd9dd7ee138e9b10ff
I20240503 02:13:03.699463 14790 ts_tablet_manager.cc:1909] T 1623a9f120c347bd9dd7ee138e9b10ff P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:13:03.702400 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table_new_name [id=cdcde9df35a2420bb80ee4a785c59fdd]) successfully deleted
I20240503 02:13:03.705737 14791 ts_tablet_manager.cc:1888] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:03.706650 14791 log.cc:1196] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/1623a9f120c347bd9dd7ee138e9b10ff
I20240503 02:13:03.707803 14791 ts_tablet_manager.cc:1909] T 1623a9f120c347bd9dd7ee138e9b10ff P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:13:03.710662 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table_new_name [id=cdcde9df35a2420bb80ee4a785c59fdd]) successfully deleted
I20240503 02:13:03.712141 14788 tablet_bootstrap.cc:654] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.724279 14787 ts_tablet_manager.cc:1888] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:03.724720 14787 log.cc:1196] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/1623a9f120c347bd9dd7ee138e9b10ff
I20240503 02:13:03.725329 14787 ts_tablet_manager.cc:1909] T 1623a9f120c347bd9dd7ee138e9b10ff P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:13:03.726742 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 1623a9f120c347bd9dd7ee138e9b10ff (table rebalancer_dynamic_table_new_name [id=cdcde9df35a2420bb80ee4a785c59fdd]) successfully deleted
W20240503 02:13:03.710608 10026 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.68:35025, user_credentials={real_user=slave}} blocked reactor thread for 71049.1us
I20240503 02:13:03.732173 14788 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:03.732395 14788 ts_tablet_manager.cc:1367] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.088s user 0.006s sys 0.000s
I20240503 02:13:03.743541 14789 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:03.744607 9949 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:03.777843 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 95ebfc29e8434ce180b7d67749aaafa0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e]), partition=
I20240503 02:13:03.777932 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 95ebfc29e8434ce180b7d67749aaafa0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e]), partition=
I20240503 02:13:03.778348 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ebfc29e8434ce180b7d67749aaafa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.778404 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ebfc29e8434ce180b7d67749aaafa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:03.770567 14651 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:03.785272 10491 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:03.812775 9943 catalog_manager.cc:5484] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:03.815727 10359 raft_consensus.cc:1261] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:13:03.816177 10026 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 83701.3us
I20240503 02:13:03.816581 14674 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:03.822120 14789 tablet_bootstrap.cc:654] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
W20240503 02:13:03.825496 10029 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.70:43707, user_credentials={real_user=slave}} blocked reactor thread for 60823us
I20240503 02:13:03.830349 14786 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:03.830662 14786 ts_tablet_manager.cc:1367] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.182s user 0.005s sys 0.000s
I20240503 02:13:03.833881 10491 raft_consensus.cc:1261] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:03.834945 14674 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:03.854959 14781 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:03.855568 14781 ts_tablet_manager.cc:1367] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.299s user 0.030s sys 0.024s
I20240503 02:13:03.857343 14781 raft_consensus.cc:348] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:03.860692 14781 raft_consensus.cc:374] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.861224 14781 raft_consensus.cc:729] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.862208 14781 consensus_queue.cc:260] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:03.869369 14781 ts_tablet_manager.cc:1398] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:13:03.871479 14781 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:03.879452 14781 tablet_bootstrap.cc:654] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.906747 14788 raft_consensus.cc:348] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:03.907097 14788 raft_consensus.cc:374] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.907219 14788 raft_consensus.cc:729] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.907490 14788 consensus_queue.cc:260] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:03.910300 13166 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:03.915151 13172 raft_consensus.cc:1261] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:03.917739 13787 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:03.919144 14788 ts_tablet_manager.cc:1398] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.184s user 0.003s sys 0.000s
W20240503 02:13:03.919461 10026 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 95519.1us
I20240503 02:13:03.941154 13787 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
W20240503 02:13:03.945387 10028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 229868us
I20240503 02:13:03.945502 10357 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:03.965936 9946 catalog_manager.cc:5484] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:03.970777 14778 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:03.971060 14778 ts_tablet_manager.cc:1367] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.452s user 0.040s sys 0.060s
I20240503 02:13:03.972172 14778 raft_consensus.cc:348] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:03.973292 14778 raft_consensus.cc:374] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:03.973496 14778 raft_consensus.cc:729] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:03.973868 14778 consensus_queue.cc:260] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:03.984594 14778 ts_tablet_manager.cc:1398] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:03.985090 14778 tablet_bootstrap.cc:492] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:03.984747 13308 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:03.987012 14778 tablet_bootstrap.cc:654] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:03.986922 10489 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:03.989200 13310 consensus_queue.cc:237] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:03.990620 13309 raft_consensus.cc:1203] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:04.013811 10224 consensus_queue.cc:237] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:04.024107 14564 tablet_copy_client.cc:308] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:04.032876 14786 raft_consensus.cc:348] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.036406 10091 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:04.041090 14786 raft_consensus.cc:374] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.043655 14786 raft_consensus.cc:729] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.045329 14786 consensus_queue.cc:260] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
W20240503 02:13:04.048017 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09a142900b5c4909ad16b4abc70ee952. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:04.054891 13174 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:04.040477 12905 consensus_queue.cc:237] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:04.023272 14816 ts_tablet_manager.cc:902] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:04.063851 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 95ebfc29e8434ce180b7d67749aaafa0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e]), partition=
I20240503 02:13:04.066354 13166 raft_consensus.cc:1261] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:04.068492 14655 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:04.071764 12907 raft_consensus.cc:1261] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:04.077370 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 95ebfc29e8434ce180b7d67749aaafa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.084692 10094 raft_consensus.cc:1261] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:04.087812 14655 consensus_queue.cc:1035] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:04.089175 14562 raft_consensus.cc:482] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.089421 14562 raft_consensus.cc:504] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:04.090260 14562 leader_election.cc:288] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:04.092775 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6056d9c9a949a6907510bd619825f6" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
W20240503 02:13:04.093665 13107 leader_election.cc:341] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.094969 14655 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:04.095863 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6056d9c9a949a6907510bd619825f6" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:04.096220 10489 raft_consensus.cc:2454] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:04.096829 13107 leader_election.cc:302] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:04.097416 14562 raft_consensus.cc:2781] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:04.097515 14562 raft_consensus.cc:482] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:04.097628 14562 raft_consensus.cc:3037] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.100785 14562 raft_consensus.cc:504] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:04.101141 10094 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:04.101619 14562 leader_election.cc:288] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:04.103900 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6056d9c9a949a6907510bd619825f6" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
W20240503 02:13:04.104851 13107 leader_election.cc:341] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.106076 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6056d9c9a949a6907510bd619825f6" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:04.106449 10489 raft_consensus.cc:3037] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.110204 10489 raft_consensus.cc:2454] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:04.111745 13107 leader_election.cc:302] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:04.112311 14562 raft_consensus.cc:2781] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:04.113078 13172 raft_consensus.cc:2932] T 559ae0e1e196444f9d50420d0f799754 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:04.125766 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 559ae0e1e196444f9d50420d0f799754 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:04.127236 14670 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:04.151717 13166 raft_consensus.cc:1261] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:04.159322 10224 raft_consensus.cc:1261] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:04.162305 10094 consensus_queue.cc:237] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:04.166651 10357 raft_consensus.cc:1261] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:04.168267 14811 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:04.170905 14720 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.018s
I20240503 02:13:04.173080 14807 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:04.180150 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9abc528ca83640c3bf66022cf2c9b914 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:04.180346 14795 consensus_queue.cc:1035] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:04.180990 13310 raft_consensus.cc:1261] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:04.181787 14807 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:04.183588 10359 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:04.183780 13308 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:04.186185 9946 catalog_manager.cc:5484] T 559ae0e1e196444f9d50420d0f799754 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:04.193353 9949 catalog_manager.cc:5484] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:04.209918 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 559ae0e1e196444f9d50420d0f799754 with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:04.215483 14778 tablet_bootstrap.cc:492] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:04.215935 14778 ts_tablet_manager.cc:1367] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.231s user 0.063s sys 0.147s
I20240503 02:13:04.217025 14778 raft_consensus.cc:348] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.217290 14778 raft_consensus.cc:374] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.217372 14778 raft_consensus.cc:729] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.217661 14778 consensus_queue.cc:260] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.219595 14778 ts_tablet_manager.cc:1398] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:04.237439 10227 consensus_queue.cc:237] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:04.240751 13174 raft_consensus.cc:1261] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:04.241830 14670 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:04.253190 14669 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:04.261792 10489 raft_consensus.cc:1261] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:04.263795 14670 consensus_queue.cc:1035] T 0dc2733b737b431684ce439c3d3fa762 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:04.266489 10489 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:04.269917 14835 tablet_replica.cc:330] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:04.270339 14835 raft_consensus.cc:2229] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:04.270884 14835 raft_consensus.cc:2258] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:04.273077 14831 tablet_bootstrap.cc:492] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:04.275576 14831 tablet_bootstrap.cc:654] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.281572 14816 tablet_copy_client.cc:308] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:04.282281 14222 raft_consensus.cc:1050] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:04.283593 14222 consensus_queue.cc:237] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:04.284636 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet e94a48cad1d648b18d709b46566d810b from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:56811)
I20240503 02:13:04.284899 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet e94a48cad1d648b18d709b46566d810b from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:56811: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-e94a48cad1d648b18d709b46566d810b
I20240503 02:13:04.289207 14795 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:04.290881 13787 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:04.298545 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e6bced8dfb104a5eb0db971cd11aad79 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:04.300894 13166 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:04.301887 9944 catalog_manager.cc:5484] T e6bced8dfb104a5eb0db971cd11aad79 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:13:04.304754 13109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.66:42397, user_credentials={real_user=slave}} blocked reactor thread for 159695us
I20240503 02:13:04.305455 10224 raft_consensus.cc:2932] T e6bced8dfb104a5eb0db971cd11aad79 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:04.309233 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0dc2733b737b431684ce439c3d3fa762 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:04.311868 9951 catalog_manager.cc:5484] T 0dc2733b737b431684ce439c3d3fa762 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:04.315125 14562 raft_consensus.cc:686] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:13:04.315532 14562 consensus_queue.cc:237] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:04.048101 13040 raft_consensus.cc:1261] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:04.318619 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet da288f76df0a409eb01ee431d3b83b49 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:49771)
I20240503 02:13:04.318848 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet da288f76df0a409eb01ee431d3b83b49 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:49771: session id = f93b103af083471b8e8c0376e9148aad-da288f76df0a409eb01ee431d3b83b49
I20240503 02:13:04.204088 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 9abc528ca83640c3bf66022cf2c9b914 with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:04.323072 14839 tablet_replica.cc:330] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:04.323362 14839 raft_consensus.cc:2229] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:04.323666 14839 raft_consensus.cc:2258] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:04.323927 13787 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:04.327644 14786 ts_tablet_manager.cc:1398] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.497s user 0.003s sys 0.000s
I20240503 02:13:04.328059 14786 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:04.330178 14839 ts_tablet_manager.cc:1875] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:04.331650 10115 tablet_copy_source_session.cc:205] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.334335 14816 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e94a48cad1d648b18d709b46566d810b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.346053 14816 tablet_copy_client.cc:783] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.346664 14816 tablet_copy_client.cc:647] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.349892 14781 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:04.350260 14781 ts_tablet_manager.cc:1367] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.479s user 0.015s sys 0.028s
I20240503 02:13:04.351342 14781 raft_consensus.cc:348] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.351739 14781 raft_consensus.cc:374] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.351887 14781 raft_consensus.cc:729] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.352298 14781 consensus_queue.cc:260] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.353127 14781 ts_tablet_manager.cc:1398] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:04.354683 10380 tablet_copy_source_session.cc:205] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.364764 12888 tablet_service.cc:1514] Processing DeleteTablet for tablet e6bced8dfb104a5eb0db971cd11aad79 with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:04.384231 14789 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:04.386162 14789 ts_tablet_manager.cc:1367] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.643s user 0.003s sys 0.003s
I20240503 02:13:04.387516 14789 raft_consensus.cc:348] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:04.387926 14789 raft_consensus.cc:374] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.388082 14789 raft_consensus.cc:729] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.388489 14789 consensus_queue.cc:260] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:04.389786 14835 ts_tablet_manager.cc:1875] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:04.399632 13174 raft_consensus.cc:2932] T 0dc2733b737b431684ce439c3d3fa762 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:04.407326 14795 raft_consensus.cc:482] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.407698 14795 raft_consensus.cc:504] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:04.412045 14795 leader_election.cc:288] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:04.330606 14786 tablet_bootstrap.cc:654] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.411962 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de58a8fe28a74c28a85c78ea8372ee0a" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:04.425212 14789 ts_tablet_manager.cc:1398] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.039s user 0.002s sys 0.000s
I20240503 02:13:04.413520 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de58a8fe28a74c28a85c78ea8372ee0a" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:04.425757 12905 raft_consensus.cc:2454] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:04.414872 14841 tablet_replica.cc:330] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:04.427500 14841 raft_consensus.cc:2229] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:13:04.427634 13246 leader_election.cc:341] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.428010 14841 raft_consensus.cc:2258] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:04.429121 14841 ts_tablet_manager.cc:1875] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:04.396570 14816 tablet_copy_client.cc:523] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:04.434731 13243 leader_election.cc:302] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:04.435161 14775 raft_consensus.cc:482] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.435334 14775 raft_consensus.cc:504] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.436314 14775 leader_election.cc:288] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:04.436554 14775 raft_consensus.cc:2781] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:04.381875 14839 ts_tablet_manager.cc:1888] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:04.436681 14775 raft_consensus.cc:482] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:04.436754 14839 log.cc:1196] T 9abc528ca83640c3bf66022cf2c9b914 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/9abc528ca83640c3bf66022cf2c9b914
I20240503 02:13:04.436775 14775 raft_consensus.cc:3037] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.437397 9949 catalog_manager.cc:5484] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 1, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:04.442099 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 9abc528ca83640c3bf66022cf2c9b914 (table rebalancer_extra_table_4 [id=b55de69d6d934d358467e9297a439a8c]) successfully deleted
I20240503 02:13:04.443321 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ebfc29e8434ce180b7d67749aaafa0" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
W20240503 02:13:04.444320 13245 leader_election.cc:341] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.446979 14779 raft_consensus.cc:482] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.447173 14779 raft_consensus.cc:504] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.448071 14779 leader_election.cc:288] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:04.448839 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ebfc29e8434ce180b7d67749aaafa0" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:04.449148 14775 raft_consensus.cc:504] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:04.449834 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e0373a72654a1082464fac8d428483" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:04.450129 10491 raft_consensus.cc:2454] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:04.450317 14775 leader_election.cc:288] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:04.452155 14789 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:04.454480 14789 tablet_bootstrap.cc:654] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.459136 14786 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:04.459496 14786 ts_tablet_manager.cc:1367] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.132s user 0.002s sys 0.003s
I20240503 02:13:04.460273 12973 leader_election.cc:302] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:04.461009 14733 raft_consensus.cc:2781] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:04.461124 14733 raft_consensus.cc:482] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:04.461267 14733 raft_consensus.cc:3037] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.463137 14786 raft_consensus.cc:348] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
W20240503 02:13:04.464742 13246 leader_election.cc:341] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.464932 13246 leader_election.cc:302] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:04.397460 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 0dc2733b737b431684ce439c3d3fa762 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:04.476702 14795 raft_consensus.cc:2726] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:04.480764 14564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet da288f76df0a409eb01ee431d3b83b49. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.481093 14848 tablet_replica.cc:330] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:04.483208 14848 raft_consensus.cc:2229] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:04.483664 14848 raft_consensus.cc:2258] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:04.489428 14848 ts_tablet_manager.cc:1875] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:04.494366 10224 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:04.495250 14668 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:04.501048 9944 catalog_manager.cc:5484] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:04.506141 13310 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:04.516265 14668 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240503 02:13:04.527875 14564 tablet_copy_client.cc:783] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.528425 14564 tablet_copy_client.cc:647] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.538558 14831 tablet_bootstrap.cc:492] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:04.538884 14831 ts_tablet_manager.cc:1367] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.267s user 0.001s sys 0.007s
W20240503 02:13:04.509487 10426 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.66:42397, user_credentials={real_user=slave}} blocked reactor thread for 80861.7us
W20240503 02:13:04.509543 10424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 86604.4us
I20240503 02:13:04.542954 14856 ts_tablet_manager.cc:902] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:04.502894 14847 raft_consensus.cc:482] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.549731 14847 raft_consensus.cc:504] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.558315 14564 tablet_copy_client.cc:523] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:04.563746 14564 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
W20240503 02:13:04.565390 10292 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 58193.8us
I20240503 02:13:04.566699 10489 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:04.464823 14816 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:04.467324 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de58a8fe28a74c28a85c78ea8372ee0a" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:04.473594 12907 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:04.474992 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e0373a72654a1082464fac8d428483" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:04.571149 12908 raft_consensus.cc:3037] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.575513 12908 raft_consensus.cc:2454] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:04.576601 13243 leader_election.cc:302] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:04.577080 14775 raft_consensus.cc:2781] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:04.577240 14775 raft_consensus.cc:686] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:04.577628 14775 consensus_queue.cc:237] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:04.581243 14856 tablet_copy_client.cc:308] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:04.585283 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 09a142900b5c4909ad16b4abc70ee952 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:35599)
I20240503 02:13:04.585490 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 09a142900b5c4909ad16b4abc70ee952 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:35599: session id = 47f901e617024baaa1865d12261ef84f-09a142900b5c4909ad16b4abc70ee952
I20240503 02:13:04.589218 14848 ts_tablet_manager.cc:1888] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:04.589401 14848 log.cc:1196] T 0dc2733b737b431684ce439c3d3fa762 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/0dc2733b737b431684ce439c3d3fa762
I20240503 02:13:04.590986 9944 catalog_manager.cc:5484] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:04.593370 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 0dc2733b737b431684ce439c3d3fa762 (table rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]) successfully deleted
I20240503 02:13:04.600244 10248 tablet_copy_source_session.cc:205] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:04.608036 14856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09a142900b5c4909ad16b4abc70ee952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:04.466356 14733 raft_consensus.cc:504] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.616374 14733 leader_election.cc:288] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:04.467532 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de58a8fe28a74c28a85c78ea8372ee0a" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:04.517784 13040 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:04.551208 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e0373a72654a1082464fac8d428483" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:04.616935 13038 raft_consensus.cc:2379] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6ec2f1283994a3089741715f811c850 in current term 1: Already voted for candidate 8d454a3607d042dc9559ed59c3ccd84b in this term.
I20240503 02:13:04.617465 14786 raft_consensus.cc:374] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.617846 14786 raft_consensus.cc:729] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.619062 14847 leader_election.cc:288] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:04.619669 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e0373a72654a1082464fac8d428483" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
W20240503 02:13:04.571763 12973 leader_election.cc:341] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:13:04.620687 12973 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.5.69:33471, user_credentials={real_user=slave}} blocked reactor thread for 48969.4us
I20240503 02:13:04.622272 14668 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:04.621930 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e0373a72654a1082464fac8d428483" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
W20240503 02:13:04.625170 12973 leader_election.cc:341] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.619536 14786 consensus_queue.cc:260] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
W20240503 02:13:04.620311 10424 leader_election.cc:341] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:04.625962 13039 raft_consensus.cc:3037] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.626451 10427 leader_election.cc:302] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ec2f1283994a3089741715f811c850; no voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:04.622287 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e0373a72654a1082464fac8d428483" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:04.626989 10489 raft_consensus.cc:3037] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.629223 13039 raft_consensus.cc:2454] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:04.629967 10489 raft_consensus.cc:2454] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:04.630553 12973 leader_election.cc:302] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:04.631001 14733 raft_consensus.cc:2781] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:04.631165 14733 raft_consensus.cc:686] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:04.631450 14846 raft_consensus.cc:2726] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:04.631520 14733 consensus_queue.cc:237] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.643178 14786 ts_tablet_manager.cc:1398] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.184s user 0.000s sys 0.002s
I20240503 02:13:04.540336 14831 raft_consensus.cc:348] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.651204 14831 raft_consensus.cc:374] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.652254 14856 tablet_copy_client.cc:783] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:04.652680 14856 tablet_copy_client.cc:647] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:04.654048 14786 tablet_bootstrap.cc:492] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:04.656394 14789 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:04.657203 14856 tablet_copy_client.cc:523] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:04.658463 14831 raft_consensus.cc:729] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.660396 14831 consensus_queue.cc:260] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.662987 14786 tablet_bootstrap.cc:654] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.663269 14668 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.668695 14789 ts_tablet_manager.cc:1367] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.217s user 0.005s sys 0.000s
I20240503 02:13:04.668752 14856 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:04.669808 14789 raft_consensus.cc:348] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.670193 14789 raft_consensus.cc:374] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.670393 14789 raft_consensus.cc:729] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.670756 14789 consensus_queue.cc:260] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:04.675024 10227 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.675730 9944 catalog_manager.cc:5484] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:04.680058 10491 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.685158 9946 catalog_manager.cc:5484] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.676744 13308 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.692639 14831 ts_tablet_manager.cc:1398] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.154s user 0.003s sys 0.000s
I20240503 02:13:04.693502 14835 ts_tablet_manager.cc:1888] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:04.693723 14835 log.cc:1196] T 559ae0e1e196444f9d50420d0f799754 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/559ae0e1e196444f9d50420d0f799754
I20240503 02:13:04.694769 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 559ae0e1e196444f9d50420d0f799754 (table rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]) successfully deleted
I20240503 02:13:04.727943 14789 ts_tablet_manager.cc:1398] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.059s user 0.002s sys 0.000s
I20240503 02:13:04.728413 14789 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:04.728538 10359 consensus_queue.cc:237] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:04.730762 14789 tablet_bootstrap.cc:654] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:04.733878 10227 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:04.733940 14856 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:04.734488 14856 tablet_bootstrap.cc:492] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:04.734829 14856 ts_tablet_manager.cc:1367] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.066s user 0.005s sys 0.004s
I20240503 02:13:04.734966 14854 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:04.735855 14856 raft_consensus.cc:348] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:04.736120 14856 raft_consensus.cc:729] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:04.736354 14856 consensus_queue.cc:260] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:04.738893 13308 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:04.739305 14854 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.740633 14668 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:04.741753 10227 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.668226 14841 ts_tablet_manager.cc:1888] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:04.749936 14841 log.cc:1196] T e6bced8dfb104a5eb0db971cd11aad79 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/e6bced8dfb104a5eb0db971cd11aad79
I20240503 02:13:04.751338 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet e6bced8dfb104a5eb0db971cd11aad79 (table rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]) successfully deleted
I20240503 02:13:04.755065 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35aea508f59f4348b05d93a5afdde4ba with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:04.758440 14856 ts_tablet_manager.cc:1398] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20240503 02:13:04.759308 10248 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-09a142900b5c4909ad16b4abc70ee952 received from {username='slave'} at 127.4.5.67:35599
I20240503 02:13:04.759480 10248 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 47f901e617024baaa1865d12261ef84f-09a142900b5c4909ad16b4abc70ee952 on tablet 09a142900b5c4909ad16b4abc70ee952 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:04.759392 9946 catalog_manager.cc:5484] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.767812 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 35aea508f59f4348b05d93a5afdde4ba with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:04.768376 14564 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:04.768400 14835 tablet_replica.cc:330] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:04.768703 14835 raft_consensus.cc:2229] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:04.768985 14564 tablet_bootstrap.cc:492] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:04.769142 14835 raft_consensus.cc:2258] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:04.769311 14564 ts_tablet_manager.cc:1367] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.206s user 0.071s sys 0.007s
I20240503 02:13:04.770321 14564 raft_consensus.cc:348] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:04.770597 14564 raft_consensus.cc:729] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:04.770805 14564 consensus_queue.cc:260] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:04.771595 14564 ts_tablet_manager.cc:1398] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20240503 02:13:04.772521 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-da288f76df0a409eb01ee431d3b83b49 received from {username='slave'} at 127.4.5.71:49771
I20240503 02:13:04.772684 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session f93b103af083471b8e8c0376e9148aad-da288f76df0a409eb01ee431d3b83b49 on tablet da288f76df0a409eb01ee431d3b83b49 with peer f93b103af083471b8e8c0376e9148aad
W20240503 02:13:04.773903 13107 consensus_peers.cc:485] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:04.776911 14562 consensus_queue.cc:1035] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:04.782377 14835 ts_tablet_manager.cc:1875] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:04.766074 13308 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:04.822809 14835 ts_tablet_manager.cc:1888] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:04.823206 14835 log.cc:1196] T 35aea508f59f4348b05d93a5afdde4ba P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/35aea508f59f4348b05d93a5afdde4ba
I20240503 02:13:04.824501 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 35aea508f59f4348b05d93a5afdde4ba (table rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]) successfully deleted
I20240503 02:13:04.825543 14786 tablet_bootstrap.cc:492] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:04.835611 14786 ts_tablet_manager.cc:1367] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.182s user 0.000s sys 0.005s
I20240503 02:13:04.836846 14786 raft_consensus.cc:348] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.837185 14786 raft_consensus.cc:374] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:04.837316 14786 raft_consensus.cc:729] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.837615 14786 consensus_queue.cc:260] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.839454 14786 ts_tablet_manager.cc:1398] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:04.854017 10227 consensus_queue.cc:237] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:04.861824 14789 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:04.862146 14789 ts_tablet_manager.cc:1367] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.134s user 0.001s sys 0.006s
I20240503 02:13:04.863083 10359 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:04.863462 14789 raft_consensus.cc:348] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:04.863693 14789 raft_consensus.cc:374] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:13:04.863979 10162 consensus_peers.cc:485] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5aa7ed62ebf44431a1a6db26ef73052a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:04.864646 13787 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:04.865254 14789 raft_consensus.cc:729] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:04.865253 10491 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:04.865581 14789 consensus_queue.cc:260] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:04.866289 14789 ts_tablet_manager.cc:1398] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:04.868393 13787 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:04.868772 14793 raft_consensus.cc:482] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:04.868989 14793 raft_consensus.cc:504] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.869927 14793 leader_election.cc:288] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:04.870815 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ebfc29e8434ce180b7d67749aaafa0" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:04.871179 13039 raft_consensus.cc:2454] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:04.872498 10029 leader_election.cc:302] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b; no voters:
I20240503 02:13:04.873951 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ebfc29e8434ce180b7d67749aaafa0" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:04.874315 13308 raft_consensus.cc:2454] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:04.876755 14793 raft_consensus.cc:2781] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:04.876916 14793 raft_consensus.cc:482] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:04.877019 14793 raft_consensus.cc:3037] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.886638 14793 raft_consensus.cc:504] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.902024 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ebfc29e8434ce180b7d67749aaafa0" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:04.902330 13308 raft_consensus.cc:3037] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.918715 13308 raft_consensus.cc:2454] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:04.919536 14793 leader_election.cc:288] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:04.920204 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "95ebfc29e8434ce180b7d67749aaafa0" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:04.920564 13039 raft_consensus.cc:3037] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:04.924021 14674 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:04.929606 10359 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:04.930804 10026 leader_election.cc:302] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:13:04.931242 14793 raft_consensus.cc:2781] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:04.931392 14793 raft_consensus.cc:686] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:13:04.931721 14793 consensus_queue.cc:237] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:04.933207 10491 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:04.939141 13039 raft_consensus.cc:2454] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:04.945822 9943 catalog_manager.cc:5484] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:04.946864 14816 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:04.947449 14816 tablet_bootstrap.cc:492] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:04.947790 14816 ts_tablet_manager.cc:1367] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.483s user 0.043s sys 0.012s
I20240503 02:13:04.948684 14816 raft_consensus.cc:348] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:04.948994 14816 raft_consensus.cc:729] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:04.949246 14816 consensus_queue.cc:260] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:04.950202 14816 ts_tablet_manager.cc:1398] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:04.951267 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-e94a48cad1d648b18d709b46566d810b received from {username='slave'} at 127.4.5.69:56811
I20240503 02:13:04.951452 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-e94a48cad1d648b18d709b46566d810b on tablet e94a48cad1d648b18d709b46566d810b with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:04.967849 9944 catalog_manager.cc:5484] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:04.975183 14775 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.001168 14775 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.018251 10359 consensus_queue.cc:237] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:05.021188 10491 raft_consensus.cc:1261] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:05.021381 10292 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 390ad47a9a534abb81d59dc7604c72d3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:05.021788 10094 raft_consensus.cc:1261] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:05.021849 14854 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:05.022439 14668 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:05.026291 14855 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:05.027740 10094 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:05.028859 10491 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:05.033463 9943 catalog_manager.cc:5484] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:05.073812 14655 raft_consensus.cc:482] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.074076 14655 raft_consensus.cc:504] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.074986 14655 leader_election.cc:288] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:05.076289 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e0373a72654a1082464fac8d428483" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:05.078603 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09e0373a72654a1082464fac8d428483" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:05.078905 10491 raft_consensus.cc:2379] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in current term 1: Already voted for candidate 8d454a3607d042dc9559ed59c3ccd84b in this term.
I20240503 02:13:05.078847 14733 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.079324 12840 leader_election.cc:302] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850
I20240503 02:13:05.079965 14655 raft_consensus.cc:3037] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.082269 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:05.083511 14655 raft_consensus.cc:2726] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:05.092319 13174 raft_consensus.cc:1203] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:05.102711 14733 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.122543 10359 raft_consensus.cc:1203] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:05.123528 13039 raft_consensus.cc:1261] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:05.128314 14793 consensus_queue.cc:1035] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.128756 13308 raft_consensus.cc:1261] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:05.129865 14793 consensus_queue.cc:1035] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:05.142107 9943 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 95ebfc29e8434ce180b7d67749aaafa0 (table rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e]). Expected version 1 got 0
I20240503 02:13:05.142769 9946 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 95ebfc29e8434ce180b7d67749aaafa0 (table rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e]). Expected version 1 got 0
I20240503 02:13:05.150499 14930 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:05.152174 14930 tablet.cc:1620] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:13:05.155123 9946 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 95ebfc29e8434ce180b7d67749aaafa0 (table rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e]). Expected version 1 got 0
I20240503 02:13:05.160423 14929 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:05.161294 14929 tablet.cc:1620] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:05.161700 9946 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 95ebfc29e8434ce180b7d67749aaafa0 (table rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e]). Expected version 1 got 0
I20240503 02:13:05.163450 14928 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:05.169592 9943 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e] alter complete (version 1)
I20240503 02:13:05.170377 14928 tablet.cc:1620] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:05.173241 14919 raft_consensus.cc:482] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.173462 14919 raft_consensus.cc:504] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:05.174405 14919 leader_election.cc:288] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:05.174999 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6056d9c9a949a6907510bd619825f6" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:05.175513 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0f6056d9c9a949a6907510bd619825f6" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:05.176797 12843 leader_election.cc:302] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:05.177474 14919 raft_consensus.cc:2726] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:05.213176 12907 raft_consensus.cc:3037] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.279839 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:05.291404 14928 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:05.292093 14928 tablet.cc:1620] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:05.294230 14929 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:05.294745 14929 tablet.cc:1620] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:05.304433 14930 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:05.305294 14930 tablet.cc:1620] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:13:05.309734 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=13d88d39700a4498a23f2823bfa4c18e] alter complete (version 2)
I20240503 02:13:05.355881 15006 ts_tablet_manager.cc:902] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:05.358052 15006 tablet_copy_client.cc:308] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:05.358918 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 5aa7ed62ebf44431a1a6db26ef73052a from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:44307)
I20240503 02:13:05.359161 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 5aa7ed62ebf44431a1a6db26ef73052a from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:44307: session id = f93b103af083471b8e8c0376e9148aad-5aa7ed62ebf44431a1a6db26ef73052a
I20240503 02:13:05.362378 10248 tablet_copy_source_session.cc:205] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:05.364084 15006 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7ed62ebf44431a1a6db26ef73052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.370658 15006 tablet_copy_client.cc:783] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:05.371481 15006 tablet_copy_client.cc:647] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:05.373685 15006 tablet_copy_client.cc:523] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:05.377375 15006 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:05.394135 10091 consensus_queue.cc:237] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
W20240503 02:13:05.407261 10026 consensus_peers.cc:485] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49ce473013a942698081c389ace54b32. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:05.408437 10359 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:05.409718 14793 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:05.410845 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_24"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:05.415313 12907 raft_consensus.cc:1203] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:05.417469 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:05.424108 10227 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:05.425770 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.425894 14793 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:05.426393 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.429796 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.430431 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.434167 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 4c206318e0524101a4daee638813e041 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.434705 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c206318e0524101a4daee638813e041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.435992 12888 tablet_service.cc:1467] Processing CreateTablet for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.436110 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 4c206318e0524101a4daee638813e041 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.436547 12888 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.436758 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 10f3ea13d971437fb3b22615df232ef5 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.437129 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f3ea13d971437fb3b22615df232ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.437007 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 10f3ea13d971437fb3b22615df232ef5 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.437598 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f3ea13d971437fb3b22615df232ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.440922 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 4c206318e0524101a4daee638813e041 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.441474 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c206318e0524101a4daee638813e041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.442309 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c206318e0524101a4daee638813e041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.444923 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 10f3ea13d971437fb3b22615df232ef5 (DEFAULT_TABLE table=rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:05.445513 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 10f3ea13d971437fb3b22615df232ef5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.451743 14928 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:05.453994 14929 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:05.458127 14930 tablet.cc:1722] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:05.462335 14792 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:05.462842 10224 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:05.492982 9949 catalog_manager.cc:5484] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:05.495404 15031 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:05.499413 15029 tablet_bootstrap.cc:492] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:05.499423 15031 tablet_bootstrap.cc:654] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.504354 15030 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:05.506728 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=13d88d39700a4498a23f2823bfa4c18e] alter complete (version 3)
I20240503 02:13:05.507084 15030 tablet_bootstrap.cc:654] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.515377 15029 tablet_bootstrap.cc:654] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.521692 15031 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:05.522009 15031 ts_tablet_manager.cc:1367] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.008s
I20240503 02:13:05.521834 10359 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:05.523332 15031 raft_consensus.cc:348] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.523775 15031 raft_consensus.cc:374] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.523970 15031 raft_consensus.cc:729] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.524370 15031 consensus_queue.cc:260] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.524847 13787 raft_consensus.cc:1050] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:05.525789 13787 consensus_queue.cc:237] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:05.526786 15031 ts_tablet_manager.cc:1398] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:05.535204 15029 tablet_bootstrap.cc:492] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:05.535651 15029 ts_tablet_manager.cc:1367] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20240503 02:13:05.536787 15029 raft_consensus.cc:348] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.537158 15029 raft_consensus.cc:374] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.537323 15029 raft_consensus.cc:729] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.537779 15029 consensus_queue.cc:260] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.539666 15029 ts_tablet_manager.cc:1398] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:05.540076 15029 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:05.540916 15027 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:05.543223 15006 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:05.543488 15027 tablet_bootstrap.cc:654] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.543787 15006 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:05.544155 15006 ts_tablet_manager.cc:1367] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.167s user 0.055s sys 0.000s
I20240503 02:13:05.544588 15028 tablet_bootstrap.cc:492] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:05.546854 15028 tablet_bootstrap.cc:654] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.548091 15006 raft_consensus.cc:348] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:05.548388 15006 raft_consensus.cc:729] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:05.548645 15006 consensus_queue.cc:260] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:05.549576 15006 ts_tablet_manager.cc:1398] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:05.550735 10248 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-5aa7ed62ebf44431a1a6db26ef73052a received from {username='slave'} at 127.4.5.71:44307
I20240503 02:13:05.550921 10248 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session f93b103af083471b8e8c0376e9148aad-5aa7ed62ebf44431a1a6db26ef73052a on tablet 5aa7ed62ebf44431a1a6db26ef73052a with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:05.550957 15032 tablet_bootstrap.cc:492] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:05.555124 15032 tablet_bootstrap.cc:654] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.559979 13039 raft_consensus.cc:1261] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:05.560520 10358 raft_consensus.cc:1261] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:05.561930 14669 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:05.562652 14669 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:05.572330 15030 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:05.572650 15030 ts_tablet_manager.cc:1367] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.068s user 0.001s sys 0.005s
I20240503 02:13:05.573690 15030 raft_consensus.cc:348] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.574071 15030 raft_consensus.cc:374] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.574256 15030 raft_consensus.cc:729] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.574604 15030 consensus_queue.cc:260] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.575860 15029 tablet_bootstrap.cc:654] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.581343 12907 raft_consensus.cc:1261] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:05.582060 14674 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:05.585224 15030 ts_tablet_manager.cc:1398] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:13:05.602989 15029 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:05.603226 15029 ts_tablet_manager.cc:1367] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.063s user 0.008s sys 0.000s
I20240503 02:13:05.604876 12908 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.607062 15029 raft_consensus.cc:348] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.607318 15029 raft_consensus.cc:374] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.607385 15029 raft_consensus.cc:729] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.607676 15029 consensus_queue.cc:260] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.608011 15028 tablet_bootstrap.cc:492] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:05.608186 15028 ts_tablet_manager.cc:1367] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.000s
I20240503 02:13:05.610325 9944 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:05.610630 9944 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:05.603860 13787 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.611594 15027 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:05.614434 15027 ts_tablet_manager.cc:1367] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.074s user 0.005s sys 0.000s
I20240503 02:13:05.615650 9944 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 95ebfc29e8434ce180b7d67749aaafa0
I20240503 02:13:05.615833 9952 catalog_manager.cc:5484] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.617213 13039 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.618219 15029 ts_tablet_manager.cc:1398] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20240503 02:13:05.618574 15029 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:05.619947 15027 raft_consensus.cc:348] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.620689 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 95ebfc29e8434ce180b7d67749aaafa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:05 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:05.621495 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 95ebfc29e8434ce180b7d67749aaafa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:05 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:05.621610 15029 tablet_bootstrap.cc:654] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.624819 13288 tablet_service.cc:1514] Processing DeleteTablet for tablet 95ebfc29e8434ce180b7d67749aaafa0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:05 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:05.633162 15029 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:05.633384 15029 ts_tablet_manager.cc:1367] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 02:13:05.621228 15028 raft_consensus.cc:348] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.633857 15028 raft_consensus.cc:374] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.634075 15028 raft_consensus.cc:729] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.634514 15029 raft_consensus.cc:348] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.634788 15029 raft_consensus.cc:374] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.634927 15029 raft_consensus.cc:729] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.635470 15025 ts_tablet_manager.cc:902] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:05.636610 15027 raft_consensus.cc:374] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.636900 15027 raft_consensus.cc:729] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.637578 15027 consensus_queue.cc:260] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.635301 15029 consensus_queue.cc:260] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.639039 15029 ts_tablet_manager.cc:1398] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:05.640204 15028 consensus_queue.cc:260] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.640993 15028 ts_tablet_manager.cc:1398] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.033s user 0.000s sys 0.003s
I20240503 02:13:05.647359 15052 tablet_replica.cc:330] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:05.648072 15055 tablet_replica.cc:330] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:05.649984 15027 ts_tablet_manager.cc:1398] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.034s user 0.002s sys 0.000s
I20240503 02:13:05.652137 15028 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:05.654263 15028 tablet_bootstrap.cc:654] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.656229 15053 tablet_replica.cc:330] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:05.656554 15053 raft_consensus.cc:2229] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:05.656883 15053 raft_consensus.cc:2258] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.658025 15053 ts_tablet_manager.cc:1875] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:05.668186 14842 raft_consensus.cc:1050] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:05.668756 10359 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.669435 14842 consensus_queue.cc:237] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:05.673046 15052 raft_consensus.cc:2229] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:05.673624 15052 raft_consensus.cc:2258] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.674860 15052 ts_tablet_manager.cc:1875] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:05.680770 10227 consensus_queue.cc:237] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:05.686367 12908 raft_consensus.cc:1261] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:05.687474 13166 raft_consensus.cc:1261] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:05.687667 10227 raft_consensus.cc:1261] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:05.687731 10094 raft_consensus.cc:1261] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:05.688198 14222 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:05.689045 14222 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:05.689694 14222 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:05.687095 13787 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.003s
I20240503 02:13:05.692915 13787 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.695562 10358 raft_consensus.cc:1261] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:05.696482 14674 consensus_queue.cc:1035] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:05.697646 12908 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.701490 10359 raft_consensus.cc:2932] T 09a142900b5c4909ad16b4abc70ee952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.706022 15032 tablet_bootstrap.cc:492] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:05.706539 15032 ts_tablet_manager.cc:1367] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.156s user 0.004s sys 0.005s
I20240503 02:13:05.708177 15055 raft_consensus.cc:2229] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:05.708580 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09a142900b5c4909ad16b4abc70ee952 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:05.709576 15032 raft_consensus.cc:348] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.709820 15032 raft_consensus.cc:374] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.709911 15032 raft_consensus.cc:729] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.710136 15032 consensus_queue.cc:260] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.710157 9944 catalog_manager.cc:5484] T 09a142900b5c4909ad16b4abc70ee952 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:05.710916 15052 ts_tablet_manager.cc:1888] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:05.711107 15052 log.cc:1196] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/95ebfc29e8434ce180b7d67749aaafa0
I20240503 02:13:05.712071 15055 raft_consensus.cc:2258] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.719244 9951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:05.720149 15053 ts_tablet_manager.cc:1888] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:05.720346 15053 log.cc:1196] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/95ebfc29e8434ce180b7d67749aaafa0
I20240503 02:13:05.721271 15032 ts_tablet_manager.cc:1398] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:13:05.722604 15055 ts_tablet_manager.cc:1875] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:05.726475 15053 ts_tablet_manager.cc:1909] T 95ebfc29e8434ce180b7d67749aaafa0 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:05.728425 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 95ebfc29e8434ce180b7d67749aaafa0 (table rebalancer_dynamic_table_new_name [id=13d88d39700a4498a23f2823bfa4c18e]) successfully deleted
I20240503 02:13:05.728682 15052 ts_tablet_manager.cc:1909] T 95ebfc29e8434ce180b7d67749aaafa0 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:13:05.729789 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 068bd8c566ca4be5b7917693650e2a28 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37]), partition=
I20240503 02:13:05.730290 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068bd8c566ca4be5b7917693650e2a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.731237 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 95ebfc29e8434ce180b7d67749aaafa0 (table rebalancer_dynamic_table_new_name [id=13d88d39700a4498a23f2823bfa4c18e]) successfully deleted
I20240503 02:13:05.733520 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 068bd8c566ca4be5b7917693650e2a28 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37]), partition=
I20240503 02:13:05.733978 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068bd8c566ca4be5b7917693650e2a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.737159 13018 tablet_service.cc:1514] Processing DeleteTablet for tablet 09a142900b5c4909ad16b4abc70ee952 with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:05.741685 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 068bd8c566ca4be5b7917693650e2a28 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37]), partition=
I20240503 02:13:05.742147 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 068bd8c566ca4be5b7917693650e2a28. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.747864 14222 raft_consensus.cc:482] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.748083 14222 raft_consensus.cc:504] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.748885 14222 leader_election.cc:288] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:05.749516 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:05.749846 12908 raft_consensus.cc:2454] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:05.751248 14733 raft_consensus.cc:482] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.751430 10292 leader_election.cc:302] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:05.751410 14733 raft_consensus.cc:504] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.751809 14222 raft_consensus.cc:2781] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.751926 14222 raft_consensus.cc:482] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.752008 14222 raft_consensus.cc:3037] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.752308 14733 leader_election.cc:288] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:05.753132 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f3ea13d971437fb3b22615df232ef5" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:05.753402 13174 raft_consensus.cc:2454] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:05.753866 12976 leader_election.cc:302] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:05.754437 14733 raft_consensus.cc:2781] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.754552 14733 raft_consensus.cc:482] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.754637 14733 raft_consensus.cc:3037] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.754549 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f3ea13d971437fb3b22615df232ef5" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:05.754776 12907 raft_consensus.cc:2454] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:05.755141 14222 raft_consensus.cc:504] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.755885 14222 leader_election.cc:288] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:05.758097 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:05.758319 12907 raft_consensus.cc:3037] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.761389 12907 raft_consensus.cc:2454] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:05.761551 15053 tablet_replica.cc:330] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:05.761976 15053 raft_consensus.cc:2229] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:05.761885 10292 leader_election.cc:302] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:05.762213 14733 raft_consensus.cc:504] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.762544 15053 raft_consensus.cc:2258] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.762616 15063 raft_consensus.cc:2781] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.763187 15063 raft_consensus.cc:686] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:13:05.763316 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f3ea13d971437fb3b22615df232ef5" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:05.763586 13174 raft_consensus.cc:3037] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.763537 15063 consensus_queue.cc:237] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:05.764425 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:05.764730 10491 raft_consensus.cc:2454] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:05.765319 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:05.765522 10491 raft_consensus.cc:3037] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.766538 13174 raft_consensus.cc:2454] T 10f3ea13d971437fb3b22615df232ef5 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:05.766981 12976 leader_election.cc:302] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:05.767292 14779 raft_consensus.cc:2781] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.768648 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "10f3ea13d971437fb3b22615df232ef5" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:05.768903 12907 raft_consensus.cc:3037] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.769101 10491 raft_consensus.cc:2454] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:05.769594 15032 tablet_bootstrap.cc:492] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:05.770658 14733 leader_election.cc:288] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:05.772595 12907 raft_consensus.cc:2454] T 10f3ea13d971437fb3b22615df232ef5 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:05.772635 13172 raft_consensus.cc:1203] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:05.775163 15053 ts_tablet_manager.cc:1875] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:05.779083 15032 tablet_bootstrap.cc:654] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.781044 15069 tablet_bootstrap.cc:492] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:05.782032 14842 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.783723 15069 tablet_bootstrap.cc:654] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.785634 13174 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.790570 10227 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.793723 9952 catalog_manager.cc:5484] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } }
W20240503 02:13:05.796470 13107 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.69:33471, user_credentials={real_user=slave}} blocked reactor thread for 112237us
I20240503 02:13:05.796739 15028 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:05.796945 15028 ts_tablet_manager.cc:1367] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.145s user 0.018s sys 0.004s
I20240503 02:13:05.797907 15028 raft_consensus.cc:348] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.798189 15028 raft_consensus.cc:374] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.798276 15028 raft_consensus.cc:729] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.798606 15028 consensus_queue.cc:260] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.799218 15028 ts_tablet_manager.cc:1398] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:05.799552 15028 tablet_bootstrap.cc:492] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:05.807567 15053 ts_tablet_manager.cc:1888] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:05.809183 15053 log.cc:1196] T 09a142900b5c4909ad16b4abc70ee952 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/09a142900b5c4909ad16b4abc70ee952
I20240503 02:13:05.811074 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 09a142900b5c4909ad16b4abc70ee952 (table rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]) successfully deleted
I20240503 02:13:05.815553 15028 tablet_bootstrap.cc:654] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:05.815847 10094 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.820820 13172 consensus_queue.cc:237] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:05.825621 9944 catalog_manager.cc:5484] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.836690 15025 tablet_copy_client.cc:308] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:05.837554 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 390ad47a9a534abb81d59dc7604c72d3 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:39119)
I20240503 02:13:05.837795 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 390ad47a9a534abb81d59dc7604c72d3 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:39119: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-390ad47a9a534abb81d59dc7604c72d3
I20240503 02:13:05.839515 15076 ts_tablet_manager.cc:902] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:05.841953 10359 consensus_queue.cc:237] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:05.843092 15032 tablet_bootstrap.cc:492] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:05.843348 15032 ts_tablet_manager.cc:1367] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.074s user 0.007s sys 0.007s
I20240503 02:13:05.844835 13174 raft_consensus.cc:1261] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:05.845450 10380 tablet_copy_source_session.cc:205] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:05.846405 15063 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:05.847534 10227 raft_consensus.cc:1261] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:05.848134 15063 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:05.850801 14222 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.853263 13174 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.856284 14919 raft_consensus.cc:482] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.856489 14919 raft_consensus.cc:504] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.857321 14919 leader_election.cc:288] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:05.857858 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b2d976f9fe42d2882b1c1ef7a01eb3" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:05.858196 13308 raft_consensus.cc:2454] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:05.858178 10224 raft_consensus.cc:2932] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.858843 12840 leader_election.cc:302] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:05.859251 14919 raft_consensus.cc:2781] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.859424 14919 raft_consensus.cc:482] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.859558 14919 raft_consensus.cc:3037] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.865191 9951 catalog_manager.cc:5484] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:05.866330 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b2d976f9fe42d2882b1c1ef7a01eb3" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:05.866689 13039 raft_consensus.cc:2454] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:05.869489 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet da288f76df0a409eb01ee431d3b83b49 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:05.870599 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet da288f76df0a409eb01ee431d3b83b49 with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:05.871050 15052 tablet_replica.cc:330] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:05.871356 15052 raft_consensus.cc:2229] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:05.871644 15052 raft_consensus.cc:2258] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:05.867983 14919 raft_consensus.cc:504] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.872525 15025 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 390ad47a9a534abb81d59dc7604c72d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:05.872794 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b2d976f9fe42d2882b1c1ef7a01eb3" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:05.873064 13308 raft_consensus.cc:3037] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.874338 15052 ts_tablet_manager.cc:1875] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:05.876240 13308 raft_consensus.cc:2454] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:05.878077 15055 ts_tablet_manager.cc:1888] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:05.879035 15069 tablet_bootstrap.cc:492] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:05.879205 15069 ts_tablet_manager.cc:1367] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.098s user 0.007s sys 0.003s
I20240503 02:13:05.879447 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90b2d976f9fe42d2882b1c1ef7a01eb3" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:05.879698 13039 raft_consensus.cc:3037] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.880098 15069 raft_consensus.cc:348] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:05.880326 15069 raft_consensus.cc:374] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.883392 13039 raft_consensus.cc:2454] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:05.884174 12843 leader_election.cc:302] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:05.885039 15080 raft_consensus.cc:2781] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:05.892266 15080 raft_consensus.cc:686] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:05.892338 14919 leader_election.cc:288] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:05.893141 15080 consensus_queue.cc:237] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.894681 15028 tablet_bootstrap.cc:492] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:05.894914 15028 ts_tablet_manager.cc:1367] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.095s user 0.003s sys 0.005s
I20240503 02:13:05.897321 15032 raft_consensus.cc:348] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:05.880427 15069 raft_consensus.cc:729] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.898737 15069 consensus_queue.cc:260] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:05.899449 15069 ts_tablet_manager.cc:1398] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240503 02:13:05.881697 15052 ts_tablet_manager.cc:1888] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:05.899771 15052 log.cc:1196] T da288f76df0a409eb01ee431d3b83b49 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/da288f76df0a409eb01ee431d3b83b49
I20240503 02:13:05.909556 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet da288f76df0a409eb01ee431d3b83b49 (table rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]) successfully deleted
I20240503 02:13:05.914310 15055 log.cc:1196] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/95ebfc29e8434ce180b7d67749aaafa0
I20240503 02:13:05.915401 14779 raft_consensus.cc:686] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:05.915798 14779 consensus_queue.cc:237] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:05.917629 15055 ts_tablet_manager.cc:1909] T 95ebfc29e8434ce180b7d67749aaafa0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:13:05.920501 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 95ebfc29e8434ce180b7d67749aaafa0 (table rebalancer_dynamic_table_new_name [id=13d88d39700a4498a23f2823bfa4c18e]) successfully deleted
I20240503 02:13:05.921588 15032 raft_consensus.cc:374] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.921922 15028 raft_consensus.cc:348] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:05.922272 15028 raft_consensus.cc:374] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:05.922395 15028 raft_consensus.cc:729] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.925135 9946 catalog_manager.cc:5484] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:05.933884 10227 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:05.935128 13308 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:05.937426 15025 tablet_copy_client.cc:783] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
W20240503 02:13:05.938374 13110 consensus_peers.cc:485] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0892c659fe1a43b796a73ee8b1783173. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:05.939167 15032 raft_consensus.cc:729] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:05.939474 15062 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.004s
I20240503 02:13:05.942962 15059 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:05.944072 15032 consensus_queue.cc:260] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:05.944795 15025 tablet_copy_client.cc:647] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:05.945209 15028 consensus_queue.cc:260] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:05.946589 15032 ts_tablet_manager.cc:1398] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.103s user 0.003s sys 0.000s
I20240503 02:13:05.947758 15025 tablet_copy_client.cc:523] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:05.951388 15028 ts_tablet_manager.cc:1398] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.056s user 0.002s sys 0.001s
I20240503 02:13:05.964761 15025 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:05.964316 15059 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:05.971169 10227 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:05.980211 9944 catalog_manager.cc:5484] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:05.989096 13308 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:05.989979 14779 raft_consensus.cc:482] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:05.990170 14779 raft_consensus.cc:504] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:05.990926 14779 leader_election.cc:288] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:05.993882 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068bd8c566ca4be5b7917693650e2a28" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:05.994233 13172 raft_consensus.cc:2454] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:05.994704 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068bd8c566ca4be5b7917693650e2a28" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:05.994984 10094 raft_consensus.cc:2454] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:05.996141 12975 leader_election.cc:302] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b; no voters:
I20240503 02:13:05.996591 14733 raft_consensus.cc:2781] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:05.996719 14733 raft_consensus.cc:482] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:05.996845 14733 raft_consensus.cc:3037] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:05.996798 9946 catalog_manager.cc:5484] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:05.999696 14733 raft_consensus.cc:504] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:06.001729 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068bd8c566ca4be5b7917693650e2a28" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:06.001984 13172 raft_consensus.cc:3037] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.003588 14733 leader_election.cc:288] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:06.005082 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "068bd8c566ca4be5b7917693650e2a28" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:06.005349 10094 raft_consensus.cc:3037] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.008162 10094 raft_consensus.cc:2454] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:06.009428 12975 leader_election.cc:302] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b; no voters:
I20240503 02:13:06.009704 14733 raft_consensus.cc:2781] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:06.011476 13172 raft_consensus.cc:2454] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:06.011360 14733 raft_consensus.cc:686] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:06.012066 14733 consensus_queue.cc:237] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:06.015611 14792 raft_consensus.cc:1050] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:06.016572 14792 consensus_queue.cc:237] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:06.031996 10227 raft_consensus.cc:1261] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.033288 14793 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.042433 10359 raft_consensus.cc:1261] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.043730 14792 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.054512 12907 raft_consensus.cc:1261] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.055219 14793 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:06.055692 14776 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.057886 12907 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:06.060362 10224 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:06.062093 10358 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:06.071841 9946 catalog_manager.cc:5484] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.077059 15025 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.077741 15025 tablet_bootstrap.cc:492] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:06.078795 15025 ts_tablet_manager.cc:1367] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.114s user 0.050s sys 0.000s
I20240503 02:13:06.080367 9951 catalog_manager.cc:5484] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.080453 15025 raft_consensus.cc:348] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:06.080874 15025 raft_consensus.cc:729] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:06.081146 15025 consensus_queue.cc:260] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:06.087021 15025 ts_tablet_manager.cc:1398] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:13:06.098340 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-390ad47a9a534abb81d59dc7604c72d3 received from {username='slave'} at 127.4.5.72:39119
I20240503 02:13:06.098505 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-390ad47a9a534abb81d59dc7604c72d3 on tablet 390ad47a9a534abb81d59dc7604c72d3 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:06.111191 15076 tablet_copy_client.cc:308] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:06.112146 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 49ce473013a942698081c389ace54b32 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:56811)
I20240503 02:13:06.112378 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 49ce473013a942698081c389ace54b32 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:56811: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-49ce473013a942698081c389ace54b32
I20240503 02:13:06.115792 10115 tablet_copy_source_session.cc:205] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:06.119190 15076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ce473013a942698081c389ace54b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.126279 15076 tablet_copy_client.cc:783] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:06.126678 15076 tablet_copy_client.cc:647] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:06.128952 15076 tablet_copy_client.cc:523] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.129144 10224 consensus_queue.cc:237] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:06.132314 10491 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:06.133353 10159 consensus_peers.cc:485] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af7beeafa08f477796e5070fbc177dcc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:06.133581 10358 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:06.133646 14674 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:06.133906 15076 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:06.134302 14669 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:06.138201 13787 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:06.138810 10491 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:06.149407 9949 catalog_manager.cc:5484] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.173730 10358 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:06.191241 15076 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.191751 15076 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:06.191828 10358 consensus_queue.cc:237] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:06.192027 15076 ts_tablet_manager.cc:1367] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.058s user 0.049s sys 0.004s
I20240503 02:13:06.192840 15076 raft_consensus.cc:348] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:06.193132 15076 raft_consensus.cc:729] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:06.193378 15076 consensus_queue.cc:260] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:06.193995 15076 ts_tablet_manager.cc:1398] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:06.194761 10094 raft_consensus.cc:1261] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:06.194859 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-49ce473013a942698081c389ace54b32 received from {username='slave'} at 127.4.5.69:56811
I20240503 02:13:06.195001 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-49ce473013a942698081c389ace54b32 on tablet 49ce473013a942698081c389ace54b32 with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:06.195286 14842 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:06.195832 10292 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5c762709f66f44b98089e69cbace1570. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:06.196942 10491 raft_consensus.cc:1261] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:06.197764 14842 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:06.199743 14222 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:06.200531 10094 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:06.202317 10491 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:06.203624 15063 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.210908 9946 catalog_manager.cc:5484] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:06.234458 14222 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.258128 12907 consensus_queue.cc:237] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:06.261054 13174 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.261193 10224 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.261972 15080 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.262516 14919 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:06.263070 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 67be061441154fbba1e227d32474cd5e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:06.263029 9949 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:06.267076 14919 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:06.267793 13174 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:06.269098 10227 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:06.273164 10094 raft_consensus.cc:1261] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:06.273736 13172 raft_consensus.cc:1261] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:06.273923 14733 consensus_queue.cc:1035] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.274121 9949 catalog_manager.cc:5484] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 4 to 5, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.275297 14733 consensus_queue.cc:1035] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.298995 14733 consensus_queue.cc:1035] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.304003 9952 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet 068bd8c566ca4be5b7917693650e2a28 (table rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37]). Expected version 1 got 0
I20240503 02:13:06.309159 15141 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:06.310261 15141 tablet.cc:1620] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:06.316079 9949 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 068bd8c566ca4be5b7917693650e2a28 (table rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37]). Expected version 1 got 0
I20240503 02:13:06.319200 9946 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 068bd8c566ca4be5b7917693650e2a28 (table rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37]). Expected version 1 got 0
I20240503 02:13:06.333621 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37] alter complete (version 1)
I20240503 02:13:06.345034 15071 consensus_queue.cc:1035] T 10f3ea13d971437fb3b22615df232ef5 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.352833 15143 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:06.355405 9949 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet 068bd8c566ca4be5b7917693650e2a28 (table rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37]). Expected version 1 got 0
I20240503 02:13:06.357134 15144 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:06.357812 15144 tablet.cc:1620] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:06.360518 15143 tablet.cc:1620] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:06.367094 10094 consensus_queue.cc:237] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
W20240503 02:13:06.371415 10029 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 476f6e414dae4db3ba9a187adad75f8c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:06.374086 10227 raft_consensus.cc:1261] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:06.374873 14793 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:06.377936 10358 raft_consensus.cc:1261] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:06.381737 12908 raft_consensus.cc:1203] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:06.384337 15100 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:06.385484 14792 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:06.387677 10224 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:06.400048 14674 raft_consensus.cc:1050] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:06.401021 14674 consensus_queue.cc:237] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:06.403798 9946 catalog_manager.cc:5484] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.410168 10358 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:06.418270 10359 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.419031 13787 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.420595 10489 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.420572 13166 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.421233 13787 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.425091 14674 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.426940 15080 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.427440 14919 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:06.443552 15155 ts_tablet_manager.cc:902] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:06.449885 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:06.464148 15143 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:06.464321 15141 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:06.464802 15143 tablet.cc:1620] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:06.464884 15141 tablet.cc:1620] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:06.476699 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=57ee0aeed4dd4107bf06f939f52fbd37] alter complete (version 2)
I20240503 02:13:06.477999 14674 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:06.488742 15144 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:06.489449 15144 tablet.cc:1620] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:06.501639 10094 consensus_queue.cc:237] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:06.504552 10359 raft_consensus.cc:1261] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:06.505293 12907 raft_consensus.cc:1261] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:06.505335 14792 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:06.505959 15097 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:06.507936 13309 raft_consensus.cc:1203] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:06.508541 15100 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:06.510406 12907 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:06.512647 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e94a48cad1d648b18d709b46566d810b with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:06.517705 9946 catalog_manager.cc:5484] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:06.520916 13172 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:06.525015 10358 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:06.530401 9949 catalog_manager.cc:5484] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
W20240503 02:13:06.552307 10161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.65:38119, user_credentials={real_user=slave}} blocked reactor thread for 133251us
I20240503 02:13:06.554733 10489 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:06.577131 15155 tablet_copy_client.cc:308] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.71:46557
I20240503 02:13:06.578553 13195 tablet_copy_service.cc:140] P f93b103af083471b8e8c0376e9148aad: Received BeginTabletCopySession request for tablet 0892c659fe1a43b796a73ee8b1783173 from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:38289)
I20240503 02:13:06.578866 13195 tablet_copy_service.cc:161] P f93b103af083471b8e8c0376e9148aad: Beginning new tablet copy session on tablet 0892c659fe1a43b796a73ee8b1783173 from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:38289: session id = 8d454a3607d042dc9559ed59c3ccd84b-0892c659fe1a43b796a73ee8b1783173
I20240503 02:13:06.580861 9952 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:06.590962 13195 tablet_copy_source_session.cc:205] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:06.601433 15155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0892c659fe1a43b796a73ee8b1783173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.606058 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:06.618229 15076 ts_tablet_manager.cc:902] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:06.619488 15144 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:06.620204 15141 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:06.622985 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 0bee269120a44b53b5bacc722f6742a9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.623541 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bee269120a44b53b5bacc722f6742a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.624370 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 0bee269120a44b53b5bacc722f6742a9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.625185 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 7afcfd9776ff4097a749ab8f91730d7b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.625552 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afcfd9776ff4097a749ab8f91730d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.626483 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 07d16e77bbe14852ae816b07379b895d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.627017 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d16e77bbe14852ae816b07379b895d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.629174 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 7afcfd9776ff4097a749ab8f91730d7b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.629663 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afcfd9776ff4097a749ab8f91730d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.629175 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 07d16e77bbe14852ae816b07379b895d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.635413 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d16e77bbe14852ae816b07379b895d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.637866 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bee269120a44b53b5bacc722f6742a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.657423 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=57ee0aeed4dd4107bf06f939f52fbd37] alter complete (version 3)
I20240503 02:13:06.662460 15172 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:06.665987 15172 tablet_bootstrap.cc:654] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.671375 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet e94a48cad1d648b18d709b46566d810b with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:06.672197 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 07d16e77bbe14852ae816b07379b895d (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.672320 10224 consensus_queue.cc:237] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:06.672761 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 07d16e77bbe14852ae816b07379b895d. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.674827 13172 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:06.675446 14669 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:06.678331 15151 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:06.682116 13172 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:06.685606 15172 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:06.685829 15172 ts_tablet_manager.cc:1367] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20240503 02:13:06.687024 15172 raft_consensus.cc:348] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:06.687445 15172 raft_consensus.cc:374] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.688829 15172 raft_consensus.cc:729] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.689357 15172 consensus_queue.cc:260] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:06.691565 15172 ts_tablet_manager.cc:1398] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:06.691934 15172 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:06.692867 10359 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:06.695686 15176 tablet_bootstrap.cc:492] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:06.696810 9946 catalog_manager.cc:5484] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:06.699762 15172 tablet_bootstrap.cc:654] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.702911 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa7ed62ebf44431a1a6db26ef73052a with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:06.703929 15176 tablet_bootstrap.cc:654] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.704337 10359 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:06.704934 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5aa7ed62ebf44431a1a6db26ef73052a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 02:13:06.708099 12975 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.65:38119, user_credentials={real_user=slave}} blocked reactor thread for 90751.1us
I20240503 02:13:06.710318 15143 tablet.cc:1722] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:06.717307 13040 consensus_queue.cc:237] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:06.728658 15179 tablet_bootstrap.cc:492] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:06.729554 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 0bee269120a44b53b5bacc722f6742a9 (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.730110 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bee269120a44b53b5bacc722f6742a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.733204 13018 tablet_service.cc:1467] Processing CreateTablet for tablet 7afcfd9776ff4097a749ab8f91730d7b (DEFAULT_TABLE table=rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:06.733650 13018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afcfd9776ff4097a749ab8f91730d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.736686 15177 tablet_replica.cc:330] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:06.737159 15172 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:06.737449 15172 ts_tablet_manager.cc:1367] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.046s user 0.003s sys 0.004s
I20240503 02:13:06.739017 15179 tablet_bootstrap.cc:654] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.739857 15177 raft_consensus.cc:2229] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:06.741829 15172 raft_consensus.cc:348] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.742646 15177 raft_consensus.cc:2258] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:06.746423 15172 raft_consensus.cc:374] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.747130 15172 raft_consensus.cc:729] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.747749 15177 ts_tablet_manager.cc:1875] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:06.748248 15172 consensus_queue.cc:260] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.751128 13787 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:06.750911 15184 ts_tablet_manager.cc:902] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:06.757222 15155 tablet_copy_client.cc:783] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:06.757557 15155 tablet_copy_client.cc:647] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
W20240503 02:13:06.758097 12973 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87bb6c6f9ea646a5a26106312a3742eb. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:06.758760 15172 ts_tablet_manager.cc:1398] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20240503 02:13:06.759299 15172 tablet_bootstrap.cc:492] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:06.759951 12908 raft_consensus.cc:1261] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:06.761595 15172 tablet_bootstrap.cc:654] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.761512 9949 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:06.761855 9949 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:06.763537 10358 raft_consensus.cc:1261] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:06.764293 15173 tablet_replica.cc:330] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:06.764657 15173 raft_consensus.cc:2229] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:06.766356 9949 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 068bd8c566ca4be5b7917693650e2a28
I20240503 02:13:06.766599 15170 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.003s
I20240503 02:13:06.767563 15173 raft_consensus.cc:2258] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:06.768013 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 068bd8c566ca4be5b7917693650e2a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:06 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:06.768733 13152 tablet_service.cc:1514] Processing DeleteTablet for tablet 068bd8c566ca4be5b7917693650e2a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:06 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:06.769353 15181 ts_tablet_manager.cc:902] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:06.770596 15173 ts_tablet_manager.cc:1875] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:06.773228 10359 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:06.774375 15182 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:06.774993 15179 tablet_bootstrap.cc:492] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:06.775295 15179 ts_tablet_manager.cc:1367] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.000s
I20240503 02:13:06.776820 15182 tablet_bootstrap.cc:654] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.778767 15179 raft_consensus.cc:348] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.779142 15179 raft_consensus.cc:374] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.779299 15179 raft_consensus.cc:729] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.781417 15179 consensus_queue.cc:260] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.783425 14733 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.017s
I20240503 02:13:06.783809 15155 tablet_copy_client.cc:523] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:06.784411 15179 ts_tablet_manager.cc:1398] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:06.786127 15172 tablet_bootstrap.cc:492] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:06.786412 15172 ts_tablet_manager.cc:1367] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.004s
I20240503 02:13:06.789089 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet 068bd8c566ca4be5b7917693650e2a28 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:06 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:06.804663 15172 raft_consensus.cc:348] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.807384 15172 raft_consensus.cc:374] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.807808 15172 raft_consensus.cc:729] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.808857 15193 tablet_replica.cc:330] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:06.809006 15172 consensus_queue.cc:260] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.809370 15193 raft_consensus.cc:2229] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:06.809787 15172 ts_tablet_manager.cc:1398] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.023s user 0.000s sys 0.002s
I20240503 02:13:06.809795 15193 raft_consensus.cc:2258] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:06.810068 15155 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:06.810393 15188 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:06.811945 15192 tablet_replica.cc:330] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:06.811928 15177 ts_tablet_manager.cc:1888] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:06.812131 15177 log.cc:1196] T 5aa7ed62ebf44431a1a6db26ef73052a P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/5aa7ed62ebf44431a1a6db26ef73052a
I20240503 02:13:06.812695 15198 tablet_replica.cc:330] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:06.812978 15198 raft_consensus.cc:2229] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:06.813450 15198 raft_consensus.cc:2258] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:06.813731 15192 raft_consensus.cc:2229] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:06.814138 15192 raft_consensus.cc:2258] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:06.814491 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 5aa7ed62ebf44431a1a6db26ef73052a (table rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]) successfully deleted
I20240503 02:13:06.815491 15192 ts_tablet_manager.cc:1875] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:06.815835 15198 ts_tablet_manager.cc:1875] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:06.817031 15100 raft_consensus.cc:482] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.817205 15100 raft_consensus.cc:504] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.816992 14733 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:06.818058 15100 leader_election.cc:288] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:06.818027 12908 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:06.819321 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d16e77bbe14852ae816b07379b895d" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
W20240503 02:13:06.820005 10028 leader_election.cc:341] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:06.823935 15182 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:06.824225 15182 ts_tablet_manager.cc:1367] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20240503 02:13:06.831094 9946 catalog_manager.cc:5484] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:06.832237 15076 tablet_copy_client.cc:308] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:06.833436 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet af7beeafa08f477796e5070fbc177dcc from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:59439)
I20240503 02:13:06.833711 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet af7beeafa08f477796e5070fbc177dcc from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:59439: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-af7beeafa08f477796e5070fbc177dcc
I20240503 02:13:06.837903 10248 tablet_copy_source_session.cc:205] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:06.838057 15182 raft_consensus.cc:348] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.840399 15076 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7beeafa08f477796e5070fbc177dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.842276 15182 raft_consensus.cc:374] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.842846 15182 raft_consensus.cc:729] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.839809 15193 ts_tablet_manager.cc:1875] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:06.846675 15182 consensus_queue.cc:260] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.848629 10357 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:06.851852 15182 ts_tablet_manager.cc:1398] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.027s user 0.003s sys 0.000s
I20240503 02:13:06.856643 15188 tablet_bootstrap.cc:654] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.870127 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:06.871462 15188 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:06.871753 15188 ts_tablet_manager.cc:1367] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.005s
I20240503 02:13:06.873353 15188 raft_consensus.cc:348] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.873703 15188 raft_consensus.cc:374] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.873876 15188 raft_consensus.cc:729] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.874274 15188 consensus_queue.cc:260] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.877239 15188 ts_tablet_manager.cc:1398] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:06.878423 15173 ts_tablet_manager.cc:1888] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:06.878715 15173 log.cc:1196] T e94a48cad1d648b18d709b46566d810b P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/e94a48cad1d648b18d709b46566d810b
I20240503 02:13:06.880173 15176 tablet_bootstrap.cc:492] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:06.879106 15059 raft_consensus.cc:482] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.880460 15176 ts_tablet_manager.cc:1367] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.185s user 0.005s sys 0.015s
I20240503 02:13:06.880640 15059 raft_consensus.cc:504] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.881635 15176 raft_consensus.cc:348] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.881865 15059 leader_election.cc:288] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:06.881906 15176 raft_consensus.cc:374] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.882081 15176 raft_consensus.cc:729] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.882421 15176 consensus_queue.cc:260] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.885020 15176 ts_tablet_manager.cc:1398] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:06.886531 13310 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d16e77bbe14852ae816b07379b895d" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:06.886967 13310 raft_consensus.cc:2454] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:06.890252 10026 leader_election.cc:302] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: eff8e8d11a4447218741bb994aa41082
I20240503 02:13:06.890663 15100 raft_consensus.cc:2781] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.890841 15100 raft_consensus.cc:482] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.890957 15100 raft_consensus.cc:3037] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.891359 14792 raft_consensus.cc:1050] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:06.892628 14792 consensus_queue.cc:237] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:06.900079 15100 raft_consensus.cc:504] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.900658 15182 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:06.903702 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet e94a48cad1d648b18d709b46566d810b (table rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]) successfully deleted
I20240503 02:13:06.904191 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d16e77bbe14852ae816b07379b895d" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:06.904693 10224 raft_consensus.cc:3037] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.909993 10337 tablet_service.cc:1467] Processing CreateTablet for tablet f9941d097fe94dd88a9abe7e3489a5bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]), partition=
I20240503 02:13:06.910650 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9941d097fe94dd88a9abe7e3489a5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.912896 10224 raft_consensus.cc:2454] T 07d16e77bbe14852ae816b07379b895d P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:06.886759 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afcfd9776ff4097a749ab8f91730d7b" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:06.913398 13040 raft_consensus.cc:2454] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:06.913615 10028 leader_election.cc:302] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:06.913610 13019 tablet_service.cc:1467] Processing CreateTablet for tablet f9941d097fe94dd88a9abe7e3489a5bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]), partition=
I20240503 02:13:06.870085 15205 ts_tablet_manager.cc:902] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:06.914000 15097 raft_consensus.cc:2781] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:06.914116 13110 leader_election.cc:302] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:06.914170 15097 raft_consensus.cc:686] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:13:06.914284 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9941d097fe94dd88a9abe7e3489a5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.914507 15097 consensus_queue.cc:237] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.914810 15059 raft_consensus.cc:2781] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:06.914955 15059 raft_consensus.cc:482] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:06.915069 15059 raft_consensus.cc:3037] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.915541 15188 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:06.917939 15188 tablet_bootstrap.cc:654] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.918866 15059 raft_consensus.cc:504] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:06.920565 10207 tablet_service.cc:1467] Processing CreateTablet for tablet f9941d097fe94dd88a9abe7e3489a5bc (DEFAULT_TABLE table=rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]), partition=
I20240503 02:13:06.920892 15182 tablet_bootstrap.cc:654] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.921052 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9941d097fe94dd88a9abe7e3489a5bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.922292 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afcfd9776ff4097a749ab8f91730d7b" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:06.922547 13040 raft_consensus.cc:3037] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.928844 13308 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afcfd9776ff4097a749ab8f91730d7b" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:06.929138 13308 raft_consensus.cc:2454] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:06.933763 15100 leader_election.cc:288] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:06.933851 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "07d16e77bbe14852ae816b07379b895d" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:06.934561 13303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7afcfd9776ff4097a749ab8f91730d7b" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:06.934846 13303 raft_consensus.cc:3037] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.936013 13302 raft_consensus.cc:3037] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:06.940114 15184 tablet_copy_client.cc:308] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:06.940552 13302 raft_consensus.cc:2454] T 07d16e77bbe14852ae816b07379b895d P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:06.941177 15218 tablet_bootstrap.cc:492] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:06.943917 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet 67be061441154fbba1e227d32474cd5e from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:52779)
I20240503 02:13:06.944085 15218 tablet_bootstrap.cc:654] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.944118 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet 67be061441154fbba1e227d32474cd5e from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:52779: session id = 47f901e617024baaa1865d12261ef84f-67be061441154fbba1e227d32474cd5e
I20240503 02:13:06.947576 12928 tablet_copy_source_session.cc:205] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:06.951761 13303 raft_consensus.cc:2454] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:06.953140 15059 leader_election.cc:288] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:06.957129 15181 tablet_copy_client.cc:308] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:06.958030 15176 tablet_bootstrap.cc:492] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:06.957968 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 5c762709f66f44b98089e69cbace1570 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:39119)
I20240503 02:13:06.958189 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 5c762709f66f44b98089e69cbace1570 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:39119: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-5c762709f66f44b98089e69cbace1570
I20240503 02:13:06.953924 15218 tablet_bootstrap.cc:492] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:06.960083 15176 tablet_bootstrap.cc:654] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:06.960285 15218 ts_tablet_manager.cc:1367] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.004s
I20240503 02:13:06.961356 15218 raft_consensus.cc:348] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:06.965103 10224 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.965931 14793 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.968663 15218 raft_consensus.cc:374] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:06.968914 15039 raft_consensus.cc:482] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.969133 15039 raft_consensus.cc:504] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:06.969942 15218 raft_consensus.cc:729] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:06.970139 10357 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.967953 15184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 67be061441154fbba1e227d32474cd5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:06.970396 15039 leader_election.cc:288] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:06.970691 14793 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:06.973281 15218 consensus_queue.cc:260] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:06.973599 9943 catalog_manager.cc:5484] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:06.973961 10380 tablet_copy_source_session.cc:205] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:06.980260 15218 ts_tablet_manager.cc:1398] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:13:06.981405 12908 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:06.985756 15155 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:06.986344 15155 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:06.986771 15155 ts_tablet_manager.cc:1367] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.191s user 0.004s sys 0.007s
I20240503 02:13:06.987226 13040 raft_consensus.cc:2454] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:06.987676 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bee269120a44b53b5bacc722f6742a9" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:06.988920 14793 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:06.989118 13246 leader_election.cc:341] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:06.989584 15155 raft_consensus.cc:348] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:06.989858 15155 raft_consensus.cc:729] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:06.990119 15155 consensus_queue.cc:260] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:06.991779 15063 raft_consensus.cc:482] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:06.991976 15063 raft_consensus.cc:504] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:06.992997 15063 leader_election.cc:288] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
W20240503 02:13:06.994315 10295 leader_election.cc:341] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Not found: Tablet not found: f9941d097fe94dd88a9abe7e3489a5bc
I20240503 02:13:06.995373 15155 ts_tablet_manager.cc:1398] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.008s user 0.002s sys 0.003s
I20240503 02:13:06.999923 15181 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c762709f66f44b98089e69cbace1570. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.001541 10094 consensus_queue.cc:237] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:07.002418 15198 ts_tablet_manager.cc:1888] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.002570 15198 log.cc:1196] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/068bd8c566ca4be5b7917693650e2a28
I20240503 02:13:07.003225 15198 ts_tablet_manager.cc:1909] T 068bd8c566ca4be5b7917693650e2a28 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:07.004165 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9941d097fe94dd88a9abe7e3489a5bc" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:07.004797 13110 leader_election.cc:302] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
W20240503 02:13:07.005152 10294 leader_election.cc:341] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:07.005376 10294 leader_election.cc:302] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47f901e617024baaa1865d12261ef84f; no voters: 8d454a3607d042dc9559ed59c3ccd84b, eff8e8d11a4447218741bb994aa41082
I20240503 02:13:07.006055 15063 raft_consensus.cc:2726] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:07.006103 15205 tablet_copy_client.cc:308] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:07.007016 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 476f6e414dae4db3ba9a187adad75f8c from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:39295)
I20240503 02:13:07.006934 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bee269120a44b53b5bacc722f6742a9" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:07.007205 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 476f6e414dae4db3ba9a187adad75f8c from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:39295: session id = 8d454a3607d042dc9559ed59c3ccd84b-476f6e414dae4db3ba9a187adad75f8c
W20240503 02:13:07.007766 13246 leader_election.cc:341] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:07.007963 13246 leader_election.cc:302] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:07.009130 15039 raft_consensus.cc:2726] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:07.009486 13195 tablet_copy_service.cc:342] P f93b103af083471b8e8c0376e9148aad: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-0892c659fe1a43b796a73ee8b1783173 received from {username='slave'} at 127.4.5.70:38289
I20240503 02:13:07.009915 14793 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.010510 10115 tablet_copy_source_session.cc:205] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:07.013567 10227 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.014432 15205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 476f6e414dae4db3ba9a187adad75f8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.016120 15184 tablet_copy_client.cc:783] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:07.016422 15184 tablet_copy_client.cc:647] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:07.020610 13303 raft_consensus.cc:1261] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.021855 12908 raft_consensus.cc:1261] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.021751 14793 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:07.025628 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 068bd8c566ca4be5b7917693650e2a28 (table rebalancer_dynamic_table_new_name [id=57ee0aeed4dd4107bf06f939f52fbd37]) successfully deleted
I20240503 02:13:07.026408 14793 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:07.027000 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 46e5690a42d44500b3e6ca24d771affc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:07.028352 14668 raft_consensus.cc:1050] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:07.029691 14668 consensus_queue.cc:237] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:07.031438 15188 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:07.031816 15188 ts_tablet_manager.cc:1367] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.116s user 0.000s sys 0.004s
I20240503 02:13:07.032899 15188 raft_consensus.cc:348] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:07.033355 15188 raft_consensus.cc:374] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.033490 15188 raft_consensus.cc:729] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.033892 15188 consensus_queue.cc:260] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:07.033887 9944 catalog_manager.cc:5484] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.039259 15188 ts_tablet_manager.cc:1398] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:07.042965 15188 tablet_bootstrap.cc:492] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:07.043349 10094 raft_consensus.cc:1261] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:07.042409 10357 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.044042 14222 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20240503 02:13:07.045351 13302 raft_consensus.cc:1261] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:07.046516 14222 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:07.046919 10490 raft_consensus.cc:1261] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:07.050143 14222 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:07.057322 15188 tablet_bootstrap.cc:654] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:07.060189 15176 tablet_bootstrap.cc:492] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:07.062314 15192 ts_tablet_manager.cc:1888] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.063056 15192 log.cc:1196] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/068bd8c566ca4be5b7917693650e2a28
I20240503 02:13:07.062985 12908 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.064553 15192 ts_tablet_manager.cc:1909] T 068bd8c566ca4be5b7917693650e2a28 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:13:07.065299 15076 tablet_copy_client.cc:783] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:07.066401 13195 tablet_copy_service.cc:433] P f93b103af083471b8e8c0376e9148aad: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-0892c659fe1a43b796a73ee8b1783173 on tablet 0892c659fe1a43b796a73ee8b1783173 with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:07.066460 15076 tablet_copy_client.cc:647] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:07.067313 15209 raft_consensus.cc:2781] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.067504 15209 raft_consensus.cc:686] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:13:07.059840 15205 tablet_copy_client.cc:783] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:07.069365 15182 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:07.069572 15182 ts_tablet_manager.cc:1367] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.181s user 0.002s sys 0.004s
I20240503 02:13:07.071019 15205 tablet_copy_client.cc:647] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:07.073426 15205 tablet_copy_client.cc:523] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:07.073762 15184 tablet_copy_client.cc:523] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:07.076726 15097 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:07.079033 15205 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:07.079933 15063 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:07.081211 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 068bd8c566ca4be5b7917693650e2a28 (table rebalancer_dynamic_table_new_name [id=57ee0aeed4dd4107bf06f939f52fbd37]) successfully deleted
I20240503 02:13:07.085681 10094 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:07.067885 15209 consensus_queue.cc:237] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:07.094823 9944 catalog_manager.cc:5484] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:07.097106 13303 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:07.100291 12907 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:07.107357 9949 catalog_manager.cc:5484] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:07.112510 13302 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:07.115545 15182 raft_consensus.cc:348] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:07.121938 15182 raft_consensus.cc:374] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.122085 15182 raft_consensus.cc:729] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.122139 10357 consensus_queue.cc:237] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:07.122426 15184 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:07.122357 15182 consensus_queue.cc:260] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:07.125455 10491 raft_consensus.cc:1261] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:07.126451 14222 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:07.129348 10491 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:07.131726 15193 ts_tablet_manager.cc:1888] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:07.131877 15193 log.cc:1196] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/068bd8c566ca4be5b7917693650e2a28
I20240503 02:13:07.132550 15193 ts_tablet_manager.cc:1909] T 068bd8c566ca4be5b7917693650e2a28 P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:13:07.133883 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 068bd8c566ca4be5b7917693650e2a28 (table rebalancer_dynamic_table_new_name [id=57ee0aeed4dd4107bf06f939f52fbd37]) successfully deleted
W20240503 02:13:07.137893 13107 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 207348us
I20240503 02:13:07.148240 15182 ts_tablet_manager.cc:1398] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.079s user 0.001s sys 0.003s
I20240503 02:13:07.160581 15076 tablet_copy_client.cc:523] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:07.181520 15076 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:07.184552 15176 ts_tablet_manager.cc:1367] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.108s user 0.000s sys 0.004s
I20240503 02:13:07.185832 15176 raft_consensus.cc:348] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:07.186203 15176 raft_consensus.cc:374] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.186312 15176 raft_consensus.cc:729] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.186684 15176 consensus_queue.cc:260] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:07.187472 15176 ts_tablet_manager.cc:1398] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:07.189424 10094 consensus_queue.cc:237] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:07.190462 14842 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:07.191418 15184 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:07.192387 12908 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:07.193197 10227 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:07.193279 14793 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:07.194944 14793 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:07.195410 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 390ad47a9a534abb81d59dc7604c72d3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:07.196224 10489 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:07.198221 14793 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.199040 9952 catalog_manager.cc:5484] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:07.199110 12908 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.201195 10227 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.204293 15184 tablet_bootstrap.cc:492] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:07.204726 15184 ts_tablet_manager.cc:1367] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.006s
I20240503 02:13:07.205888 15184 raft_consensus.cc:348] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:07.206233 15184 raft_consensus.cc:729] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:07.207006 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 390ad47a9a534abb81d59dc7604c72d3 with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:07.207459 15192 tablet_replica.cc:330] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:07.207805 15192 raft_consensus.cc:2229] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.208586 15192 raft_consensus.cc:2258] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.208945 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49ce473013a942698081c389ace54b32 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:07.210112 15192 ts_tablet_manager.cc:1875] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.210544 9946 catalog_manager.cc:5484] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:07.218370 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ce473013a942698081c389ace54b32 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:07.219769 13303 raft_consensus.cc:1261] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:07.220799 14842 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:07.223222 13303 raft_consensus.cc:2932] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:07.224138 15181 tablet_copy_client.cc:783] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:07.227523 15192 ts_tablet_manager.cc:1888] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:07.227849 15192 log.cc:1196] T 390ad47a9a534abb81d59dc7604c72d3 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/390ad47a9a534abb81d59dc7604c72d3
I20240503 02:13:07.206658 15184 consensus_queue.cc:260] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:07.230439 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 390ad47a9a534abb81d59dc7604c72d3 (table rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]) successfully deleted
I20240503 02:13:07.231199 15188 tablet_bootstrap.cc:492] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:07.231405 15188 ts_tablet_manager.cc:1367] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.189s user 0.011s sys 0.006s
I20240503 02:13:07.232434 15188 raft_consensus.cc:348] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:07.232862 15188 raft_consensus.cc:374] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:07.233481 15188 raft_consensus.cc:729] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:07.234351 15188 consensus_queue.cc:260] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:07.235160 15184 ts_tablet_manager.cc:1398] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240503 02:13:07.235321 9952 catalog_manager.cc:5484] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 1, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:07.236039 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-67be061441154fbba1e227d32474cd5e received from {username='slave'} at 127.4.5.67:52779
I20240503 02:13:07.236284 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session 47f901e617024baaa1865d12261ef84f-67be061441154fbba1e227d32474cd5e on tablet 67be061441154fbba1e227d32474cd5e with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:07.236864 15188 ts_tablet_manager.cc:1398] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:07.250789 15250 tablet_replica.cc:330] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:07.251204 15250 raft_consensus.cc:2229] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:07.251613 15250 raft_consensus.cc:2258] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:07.253062 15250 ts_tablet_manager.cc:1875] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:07.255573 15181 tablet_copy_client.cc:647] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:07.258033 15252 ts_tablet_manager.cc:902] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:07.260715 15181 tablet_copy_client.cc:523] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:07.270437 15181 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:07.272892 10359 raft_consensus.cc:1203] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:13:07.294095 15250 ts_tablet_manager.cc:1888] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:07.294718 15250 log.cc:1196] T 49ce473013a942698081c389ace54b32 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/49ce473013a942698081c389ace54b32
I20240503 02:13:07.296814 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 49ce473013a942698081c389ace54b32 (table rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]) successfully deleted
I20240503 02:13:07.333842 13039 consensus_queue.cc:237] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:07.342625 15231 consensus_queue.cc:1035] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.346860 14792 consensus_queue.cc:1035] T 07d16e77bbe14852ae816b07379b895d P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:07.349273 12975 consensus_peers.cc:485] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e3bba75394f24d4c91562089abad24c4. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:07.350636 13172 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.351389 15170 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:07.355652 15240 raft_consensus.cc:482] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.355830 15240 raft_consensus.cc:504] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:07.356649 15240 leader_election.cc:288] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:07.357474 13303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bee269120a44b53b5bacc722f6742a9" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:07.357828 13303 raft_consensus.cc:2454] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:07.358983 13107 leader_election.cc:302] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:07.361493 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bee269120a44b53b5bacc722f6742a9" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:07.361842 13040 raft_consensus.cc:2454] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:07.363498 15240 raft_consensus.cc:2781] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:07.363649 15240 raft_consensus.cc:482] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:07.363742 15240 raft_consensus.cc:3037] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.367180 15240 raft_consensus.cc:504] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:07.368921 13303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bee269120a44b53b5bacc722f6742a9" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:07.369192 13303 raft_consensus.cc:3037] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.375216 15170 raft_consensus.cc:482] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:07.375555 13303 raft_consensus.cc:2454] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:07.375456 15170 raft_consensus.cc:504] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:07.376122 13107 leader_election.cc:302] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:07.376492 15209 raft_consensus.cc:2781] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.376677 15209 raft_consensus.cc:686] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:13:07.376977 15209 consensus_queue.cc:237] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:07.377998 15170 leader_election.cc:288] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:07.379137 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9941d097fe94dd88a9abe7e3489a5bc" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:13:07.379391 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9941d097fe94dd88a9abe7e3489a5bc" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:07.379534 10359 raft_consensus.cc:2454] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:07.379685 10224 raft_consensus.cc:2454] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:07.380089 12975 leader_election.cc:302] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, 8d454a3607d042dc9559ed59c3ccd84b; no voters:
I20240503 02:13:07.380560 15170 raft_consensus.cc:2781] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:07.380700 15170 raft_consensus.cc:482] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:07.380811 15170 raft_consensus.cc:3037] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.392835 15240 leader_election.cc:288] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:07.394021 15170 raft_consensus.cc:504] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:07.397135 15170 leader_election.cc:288] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:07.397684 10359 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9941d097fe94dd88a9abe7e3489a5bc" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:07.397971 10359 raft_consensus.cc:3037] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.401553 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9941d097fe94dd88a9abe7e3489a5bc" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:07.401875 10224 raft_consensus.cc:3037] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.406253 13305 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.406842 10224 raft_consensus.cc:2454] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:07.407446 12975 leader_election.cc:302] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:07.407773 14733 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:07.408807 14733 raft_consensus.cc:2781] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:07.408931 14733 raft_consensus.cc:686] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:07.409198 14733 consensus_queue.cc:237] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:07.411578 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0bee269120a44b53b5bacc722f6742a9" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:07.411850 13039 raft_consensus.cc:3037] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:07.412272 10359 raft_consensus.cc:2454] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:07.416044 13039 raft_consensus.cc:2454] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:07.418340 9952 catalog_manager.cc:5484] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 1, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:07.437052 15252 tablet_copy_client.cc:308] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:07.438220 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet 87bb6c6f9ea646a5a26106312a3742eb from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:38469)
I20240503 02:13:07.438437 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet 87bb6c6f9ea646a5a26106312a3742eb from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:38469: session id = c6ec2f1283994a3089741715f811c850-87bb6c6f9ea646a5a26106312a3742eb
I20240503 02:13:07.441689 13061 tablet_copy_source_session.cc:205] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:07.446543 15170 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:07.446729 9952 catalog_manager.cc:5484] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:07.448336 13302 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:07.464236 15076 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:07.465737 15252 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 87bb6c6f9ea646a5a26106312a3742eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.469385 15076 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:07.469779 15076 ts_tablet_manager.cc:1367] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.288s user 0.063s sys 0.000s
I20240503 02:13:07.471009 15076 raft_consensus.cc:348] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:07.471385 15076 raft_consensus.cc:729] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:07.471649 15076 consensus_queue.cc:260] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:07.473742 15076 ts_tablet_manager.cc:1398] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:07.475003 10248 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-af7beeafa08f477796e5070fbc177dcc received from {username='slave'} at 127.4.5.69:59439
I20240503 02:13:07.475210 10248 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-af7beeafa08f477796e5070fbc177dcc on tablet af7beeafa08f477796e5070fbc177dcc with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:07.475553 13172 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:07.487656 13039 raft_consensus.cc:1203] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:07.502656 15252 tablet_copy_client.cc:783] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:07.503420 15252 tablet_copy_client.cc:647] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:07.506964 15252 tablet_copy_client.cc:523] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:07.513998 9949 catalog_manager.cc:5484] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:07.532342 15252 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:07.556972 15252 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:07.557632 15252 tablet_bootstrap.cc:492] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:07.557962 15252 ts_tablet_manager.cc:1367] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.009s
I20240503 02:13:07.558177 15205 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:07.558897 15205 tablet_bootstrap.cc:492] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:07.559329 15205 ts_tablet_manager.cc:1367] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.480s user 0.062s sys 0.003s
I20240503 02:13:07.559207 15252 raft_consensus.cc:348] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:07.559602 15252 raft_consensus.cc:729] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:07.559904 15252 consensus_queue.cc:260] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:07.560942 15252 ts_tablet_manager.cc:1398] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:07.564107 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-87bb6c6f9ea646a5a26106312a3742eb received from {username='slave'} at 127.4.5.68:38469
I20240503 02:13:07.564272 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session c6ec2f1283994a3089741715f811c850-87bb6c6f9ea646a5a26106312a3742eb on tablet 87bb6c6f9ea646a5a26106312a3742eb with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:07.569861 15205 raft_consensus.cc:348] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:07.570204 15205 raft_consensus.cc:729] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:07.578907 15209 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.588699 15181 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:07.589509 15181 tablet_bootstrap.cc:492] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:07.589941 15181 ts_tablet_manager.cc:1367] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.320s user 0.049s sys 0.007s
I20240503 02:13:07.593559 15181 raft_consensus.cc:348] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:07.594058 15181 raft_consensus.cc:729] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:07.594383 15181 consensus_queue.cc:260] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:07.596114 15184 ts_tablet_manager.cc:902] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:07.596396 15181 ts_tablet_manager.cc:1398] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:07.597271 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-5c762709f66f44b98089e69cbace1570 received from {username='slave'} at 127.4.5.72:39119
I20240503 02:13:07.597433 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-5c762709f66f44b98089e69cbace1570 on tablet 5c762709f66f44b98089e69cbace1570 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:07.570453 15205 consensus_queue.cc:260] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:07.598762 15205 ts_tablet_manager.cc:1398] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20240503 02:13:07.599623 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-476f6e414dae4db3ba9a187adad75f8c received from {username='slave'} at 127.4.5.70:39295
I20240503 02:13:07.599803 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-476f6e414dae4db3ba9a187adad75f8c on tablet 476f6e414dae4db3ba9a187adad75f8c with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:07.608726 15059 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.627265 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:07.640895 15184 tablet_copy_client.cc:308] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:07.641860 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 46e5690a42d44500b3e6ca24d771affc from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:60077)
I20240503 02:13:07.642246 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 46e5690a42d44500b3e6ca24d771affc from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:60077: session id = 47f901e617024baaa1865d12261ef84f-46e5690a42d44500b3e6ca24d771affc
I20240503 02:13:07.651424 10115 tablet_copy_source_session.cc:205] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:07.662494 15184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 46e5690a42d44500b3e6ca24d771affc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.664244 10227 raft_consensus.cc:1261] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.665071 14733 consensus_queue.cc:1035] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.665086 13040 consensus_queue.cc:237] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:07.674594 13172 raft_consensus.cc:1261] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:07.675839 12975 consensus_peers.cc:485] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5968b9989fb4f37bb4422645e7c8032. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:07.675874 12908 raft_consensus.cc:1261] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.679350 10359 raft_consensus.cc:1261] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:07.679948 15270 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:07.679999 9944 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]). Expected version 1 got 0
I20240503 02:13:07.680788 15270 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:07.681104 15184 tablet_copy_client.cc:783] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:07.681375 15184 tablet_copy_client.cc:647] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:07.683022 14733 consensus_queue.cc:1035] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.686280 15290 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:07.688839 9944 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]). Expected version 1 got 0
I20240503 02:13:07.691125 15290 tablet.cc:1620] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:07.691629 15184 tablet_copy_client.cc:523] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:07.699790 15289 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:07.700424 15289 tablet.cc:1620] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:07.700557 15184 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:07.702008 15293 mvcc.cc:205] Tried to move back new op lower bound from 7023420979844837376 to 7023420978830393344. Current Snapshot: MvccSnapshot[applied={T|T < 7023420979844837376}]
I20240503 02:13:07.710100 15170 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:07.713290 9944 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078] alter complete (version 1)
I20240503 02:13:07.714665 12908 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:07.715667 15184 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:07.716094 15184 tablet_bootstrap.cc:492] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:07.716356 15184 ts_tablet_manager.cc:1367] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20240503 02:13:07.717188 15184 raft_consensus.cc:348] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:07.728790 9943 catalog_manager.cc:5484] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:07.754334 13172 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:07.717417 15184 raft_consensus.cc:729] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:07.759884 15184 consensus_queue.cc:260] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:07.764101 15184 ts_tablet_manager.cc:1398] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.048s user 0.004s sys 0.000s
I20240503 02:13:07.765568 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-46e5690a42d44500b3e6ca24d771affc received from {username='slave'} at 127.4.5.67:60077
I20240503 02:13:07.765745 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session 47f901e617024baaa1865d12261ef84f-46e5690a42d44500b3e6ca24d771affc on tablet 46e5690a42d44500b3e6ca24d771affc with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:07.771579 15305 ts_tablet_manager.cc:902] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:07.779871 10489 raft_consensus.cc:1203] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:07.793061 13302 raft_consensus.cc:1203] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:07.824398 12907 raft_consensus.cc:1203] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:07.825721 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:07.832924 15296 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:07.834730 15296 tablet.cc:1620] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: MemRowSet was empty: no flush needed.
I20240503 02:13:07.844830 9944 catalog_manager.cc:5576] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899) does not have the latest schema for tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]). Expected version 2 got 0
I20240503 02:13:07.850885 9944 catalog_manager.cc:5576] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899) does not have the latest schema for tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]). Expected version 2 got 1
I20240503 02:13:07.858464 13039 raft_consensus.cc:1203] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:07.859977 15081 raft_consensus.cc:1050] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:07.861155 15081 consensus_queue.cc:237] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:07.861793 9943 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]). Expected version 2 got 1
I20240503 02:13:07.862708 15289 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:07.863327 15289 tablet.cc:1620] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:07.881357 15296 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:07.881987 15296 tablet.cc:1620] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: MemRowSet was empty: no flush needed.
I20240503 02:13:07.881999 9944 catalog_manager.cc:5576] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899) does not have the latest schema for tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078]). Expected version 2 got 1
I20240503 02:13:07.893020 9944 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=a9303d04887745fdb936c571ea51f078] alter complete (version 2)
I20240503 02:13:07.906548 15305 tablet_copy_client.cc:308] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:07.907511 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet e3bba75394f24d4c91562089abad24c4 from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:40827)
I20240503 02:13:07.907613 10357 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:07.907740 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet e3bba75394f24d4c91562089abad24c4 from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:40827: session id = 031a0afb01b6469299fa829fa3467ef7-e3bba75394f24d4c91562089abad24c4
I20240503 02:13:07.910342 15311 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:07.910812 13061 tablet_copy_source_session.cc:205] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:07.912453 10224 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:07.914582 15311 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:07.914678 15290 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:07.915285 15290 tablet.cc:1620] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:07.935649 15311 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:07.939234 15305 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e3bba75394f24d4c91562089abad24c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:07.942593 10227 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:07.951737 9943 catalog_manager.cc:5484] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 5 to 6, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:07.954821 15209 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:07.961267 9943 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_28"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:07.965476 10489 consensus_queue.cc:237] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:07.967329 15059 raft_consensus.cc:1050] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:07.969017 15059 consensus_queue.cc:237] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:07.980669 15209 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:07.984788 10424 consensus_peers.cc:485] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 31f72bc850c74289aa3a59ecc3dcc287. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:07.989691 10227 raft_consensus.cc:1261] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:07.990427 15048 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:07.990705 10357 raft_consensus.cc:1261] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:07.991760 15048 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:07.996311 15048 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:07.997325 10227 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:08.003345 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:08.011883 13019 tablet_service.cc:1467] Processing CreateTablet for tablet e8cf65124d3a4ccd8eded963fb7e92e8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.012524 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cf65124d3a4ccd8eded963fb7e92e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.012665 12886 tablet_service.cc:1467] Processing CreateTablet for tablet e8cf65124d3a4ccd8eded963fb7e92e8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.013219 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cf65124d3a4ccd8eded963fb7e92e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.014569 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 4c014b2069ae42788aae87d8304fa155 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.015233 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c014b2069ae42788aae87d8304fa155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.016033 13289 tablet_service.cc:1467] Processing CreateTablet for tablet e8cf65124d3a4ccd8eded963fb7e92e8 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.016722 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cf65124d3a4ccd8eded963fb7e92e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.018723 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 44a129a73db84b74af353de0ec089c93 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.019210 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a129a73db84b74af353de0ec089c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.020298 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 4c014b2069ae42788aae87d8304fa155 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.021207 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 4c014b2069ae42788aae87d8304fa155 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.021633 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c014b2069ae42788aae87d8304fa155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.026427 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 44a129a73db84b74af353de0ec089c93 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.026906 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 44a129a73db84b74af353de0ec089c93 (DEFAULT_TABLE table=rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:08.027200 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a129a73db84b74af353de0ec089c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.027401 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 44a129a73db84b74af353de0ec089c93. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.030128 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c014b2069ae42788aae87d8304fa155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.047765 15289 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:08.048926 15290 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:08.053222 15296 tablet.cc:1722] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:08.067358 10359 raft_consensus.cc:1203] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:08.070405 13040 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.074967 10224 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.075893 15243 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.076721 15243 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.088402 13302 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.090174 15243 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.099457 10357 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:08.109028 15209 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.117762 13039 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.118034 13302 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.123859 15339 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:08.135711 9950 catalog_manager.cc:5484] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.136415 15333 tablet_bootstrap.cc:492] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:08.139328 15333 tablet_bootstrap.cc:654] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.139842 15339 tablet_bootstrap.cc:654] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.145884 15305 tablet_copy_client.cc:783] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:08.146414 15305 tablet_copy_client.cc:647] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
W20240503 02:13:08.147516 12842 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.66:42397, user_credentials={real_user=slave}} blocked reactor thread for 199248us
I20240503 02:13:08.151655 9949 catalog_manager.cc:5484] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.170636 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=a9303d04887745fdb936c571ea51f078] alter complete (version 3)
I20240503 02:13:08.174567 13172 consensus_queue.cc:237] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:08.178375 15305 tablet_copy_client.cc:523] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:08.179504 13040 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:08.180284 15243 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:08.185261 15240 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.186806 13039 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.189473 13302 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:08.191186 15059 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:08.193668 13302 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.195905 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0892c659fe1a43b796a73ee8b1783173 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:08.202031 9943 catalog_manager.cc:5484] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:08.223126 15305 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:08.225881 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 0892c659fe1a43b796a73ee8b1783173 with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:08.240211 15349 ts_tablet_manager.cc:902] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:08.246635 15063 raft_consensus.cc:1050] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:08.248013 15063 consensus_queue.cc:237] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:08.252550 15350 tablet_replica.cc:330] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:08.254014 15350 raft_consensus.cc:2229] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.254653 15350 pending_rounds.cc:62] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Trying to abort 1 pending ops.
I20240503 02:13:08.255331 15350 pending_rounds.cc:68] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023420981117100032 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "0892c659fe1a43b796a73ee8b1783173" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } } }
I20240503 02:13:08.256086 15350 raft_consensus.cc:2894] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 02:13:08.256304 15350 raft_consensus.cc:2866] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:08.256980 15350 raft_consensus.cc:2258] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.263136 15336 tablet_bootstrap.cc:492] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
W20240503 02:13:08.263669 15335 log.cc:924] Time spent T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Append to log took a long time: real 0.112s user 0.000s sys 0.000s
W20240503 02:13:08.270350 15335 consensus_queue.cc:1159] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "eff8e8d11a4447218741bb994aa41082" status { last_received { term: 1 index: 3 } last_committed_idx: 2 last_received_current_leader { term: 1 index: 3 } }
I20240503 02:13:08.274627 15339 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:08.274986 15339 ts_tablet_manager.cc:1367] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.054s
I20240503 02:13:08.276064 15339 raft_consensus.cc:348] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.276422 15339 raft_consensus.cc:374] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.276618 15339 raft_consensus.cc:729] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.264998 15332 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:08.277318 15339 consensus_queue.cc:260] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.279232 15332 tablet_bootstrap.cc:654] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.279927 15339 ts_tablet_manager.cc:1398] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:08.280831 15339 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:08.283107 15339 tablet_bootstrap.cc:654] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.290830 15339 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:08.291100 15339 ts_tablet_manager.cc:1367] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20240503 02:13:08.292102 15339 raft_consensus.cc:348] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.292434 15339 raft_consensus.cc:374] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.292634 15339 raft_consensus.cc:729] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.293018 15339 consensus_queue.cc:260] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.293828 15339 ts_tablet_manager.cc:1398] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:08.295130 10094 raft_consensus.cc:1261] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.296343 15336 tablet_bootstrap.cc:654] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.296475 10489 raft_consensus.cc:1261] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.297775 15330 tablet_bootstrap.cc:492] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:08.298288 13303 raft_consensus.cc:1261] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.299918 15330 tablet_bootstrap.cc:654] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.279690 15350 ts_tablet_manager.cc:1875] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:08.302325 13172 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:08.302196 10359 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.303498 15301 raft_consensus.cc:1050] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:08.304564 15301 consensus_queue.cc:237] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:08.306613 15351 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.309115 9947 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:08.309798 9947 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:08.322547 15333 tablet_bootstrap.cc:492] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:08.323282 15333 ts_tablet_manager.cc:1367] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.187s user 0.003s sys 0.015s
I20240503 02:13:08.325049 15324 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.019s
I20240503 02:13:08.325335 9947 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet f9941d097fe94dd88a9abe7e3489a5bc
I20240503 02:13:08.329412 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet f9941d097fe94dd88a9abe7e3489a5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:08 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:08.330396 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet f9941d097fe94dd88a9abe7e3489a5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:08 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:08.334887 15333 raft_consensus.cc:348] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.335860 15333 raft_consensus.cc:374] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.336550 15354 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:08.344432 15333 raft_consensus.cc:729] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.350026 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet f9941d097fe94dd88a9abe7e3489a5bc with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:08 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:08.350682 15170 raft_consensus.cc:1050] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:08.351500 15170 consensus_queue.cc:237] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:08.353971 15330 tablet_bootstrap.cc:492] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:08.354178 15330 ts_tablet_manager.cc:1367] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20240503 02:13:08.345857 15333 consensus_queue.cc:260] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.358872 15333 ts_tablet_manager.cc:1398] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.035s user 0.002s sys 0.000s
I20240503 02:13:08.361959 15063 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.366829 15271 raft_consensus.cc:482] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.367090 15271 raft_consensus.cc:504] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.368114 15271 leader_election.cc:288] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:08.368620 15330 raft_consensus.cc:348] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.368986 15354 tablet_bootstrap.cc:654] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.368867 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c014b2069ae42788aae87d8304fa155" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:08.370110 15330 raft_consensus.cc:374] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.370306 15330 raft_consensus.cc:729] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.372148 15231 raft_consensus.cc:1050] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:08.373091 15231 consensus_queue.cc:237] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:08.370617 15330 consensus_queue.cc:260] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.378343 15330 ts_tablet_manager.cc:1398] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20240503 02:13:08.378728 15330 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:08.383589 15350 ts_tablet_manager.cc:1888] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:08.383874 15350 log.cc:1196] T 0892c659fe1a43b796a73ee8b1783173 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/0892c659fe1a43b796a73ee8b1783173
I20240503 02:13:08.385788 13039 raft_consensus.cc:1261] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.386566 14793 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.392827 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 0892c659fe1a43b796a73ee8b1783173 (table rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]) successfully deleted
I20240503 02:13:08.394675 15209 raft_consensus.cc:482] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.394989 15209 raft_consensus.cc:504] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.395465 15350 tablet_replica.cc:330] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:08.396129 15209 leader_election.cc:288] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:08.396270 15350 raft_consensus.cc:2229] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.396631 15350 raft_consensus.cc:2258] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.397779 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a129a73db84b74af353de0ec089c93" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:08.398124 10091 raft_consensus.cc:2454] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:08.399276 13109 leader_election.cc:302] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:08.399860 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a129a73db84b74af353de0ec089c93" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
W20240503 02:13:08.400660 13110 leader_election.cc:341] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:08.401237 15209 raft_consensus.cc:2781] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.401357 15209 raft_consensus.cc:482] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.401446 15209 raft_consensus.cc:3037] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.405251 15209 raft_consensus.cc:504] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.405393 15345 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240503 02:13:08.406600 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a129a73db84b74af353de0ec089c93" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
W20240503 02:13:08.407254 13110 leader_election.cc:341] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:08.408397 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a129a73db84b74af353de0ec089c93" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:08.408782 10091 raft_consensus.cc:3037] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.412930 10091 raft_consensus.cc:2454] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:08.413532 13109 leader_election.cc:302] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters: 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:08.413962 15059 raft_consensus.cc:2781] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.414152 15059 raft_consensus.cc:686] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:13:08.414624 15336 tablet_bootstrap.cc:492] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:08.414556 15059 consensus_queue.cc:237] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.414796 15336 ts_tablet_manager.cc:1367] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.271s user 0.003s sys 0.015s
I20240503 02:13:08.415781 15336 raft_consensus.cc:348] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.416014 15336 raft_consensus.cc:374] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.416121 15336 raft_consensus.cc:729] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.416363 15336 consensus_queue.cc:260] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.417269 15209 leader_election.cc:288] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:08.419848 15350 ts_tablet_manager.cc:1875] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:08.419685 13172 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.422765 10359 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.425983 15365 tablet_replica.cc:330] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:08.426415 15365 raft_consensus.cc:2229] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.426744 15365 raft_consensus.cc:2258] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.426937 12908 consensus_queue.cc:237] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:08.419457 15367 tablet_replica.cc:330] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:08.429523 15367 raft_consensus.cc:2229] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:08.430002 15367 raft_consensus.cc:2258] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.431422 15367 ts_tablet_manager.cc:1875] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:13:08.431800 13245 leader_election.cc:341] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:08.432729 10359 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:08.433319 15336 ts_tablet_manager.cc:1398] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.018s user 0.003s sys 0.000s
I20240503 02:13:08.433467 15324 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:08.433804 15336 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:08.435700 15336 tablet_bootstrap.cc:654] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.436206 12905 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.436952 15365 ts_tablet_manager.cc:1875] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:08.438144 10358 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.446518 9951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:08.449707 15367 ts_tablet_manager.cc:1888] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:08.449204 15324 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.451848 15330 tablet_bootstrap.cc:654] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.457015 15367 log.cc:1196] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/f9941d097fe94dd88a9abe7e3489a5bc
I20240503 02:13:08.457105 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c014b2069ae42788aae87d8304fa155" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:08.457907 15367 ts_tablet_manager.cc:1909] T f9941d097fe94dd88a9abe7e3489a5bc P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:08.462198 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table_new_name [id=a9303d04887745fdb936c571ea51f078]) successfully deleted
I20240503 02:13:08.466013 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 67be061441154fbba1e227d32474cd5e with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:08.472366 15305 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 02:13:08.472823 13245 leader_election.cc:341] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:08.473310 15305 tablet_bootstrap.cc:492] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:08.473284 13245 leader_election.cc:302] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082
I20240503 02:13:08.473696 15271 raft_consensus.cc:2726] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:08.479238 9943 catalog_manager.cc:5484] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 6 to 7, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:08.482158 15354 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:08.482460 15354 ts_tablet_manager.cc:1367] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.146s user 0.009s sys 0.000s
I20240503 02:13:08.484836 15305 ts_tablet_manager.cc:1367] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.262s user 0.008s sys 0.010s
I20240503 02:13:08.487496 13152 tablet_service.cc:1514] Processing DeleteTablet for tablet 67be061441154fbba1e227d32474cd5e with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:08.495805 10490 consensus_queue.cc:237] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:08.503208 9950 catalog_manager.cc:5484] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 1, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.505718 15382 tablet_replica.cc:330] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:08.506175 15382 raft_consensus.cc:2229] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.506614 15382 raft_consensus.cc:2258] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.507488 15336 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:08.507805 15336 ts_tablet_manager.cc:1367] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.074s user 0.001s sys 0.015s
I20240503 02:13:08.507826 15382 ts_tablet_manager.cc:1875] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:08.509109 10491 raft_consensus.cc:1261] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.509516 10359 raft_consensus.cc:1261] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.510335 15270 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.511299 14733 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.511819 15336 raft_consensus.cc:348] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.512061 15336 raft_consensus.cc:374] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.512573 15336 raft_consensus.cc:729] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.513062 15336 consensus_queue.cc:260] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.515117 15305 raft_consensus.cc:348] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:08.515486 15305 raft_consensus.cc:729] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:08.515830 15305 consensus_queue.cc:260] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
W20240503 02:13:08.516081 10427 consensus_peers.cc:485] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dd081871c430470b95fce8f19634dc2e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:08.516752 15305 ts_tablet_manager.cc:1398] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:08.518774 13302 raft_consensus.cc:1261] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:08.519618 15048 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:08.518298 12908 raft_consensus.cc:1261] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:08.483553 15354 raft_consensus.cc:348] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.521597 15354 raft_consensus.cc:374] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.521734 15354 raft_consensus.cc:729] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.521852 15365 ts_tablet_manager.cc:1888] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:08.522015 15365 log.cc:1196] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/f9941d097fe94dd88a9abe7e3489a5bc
I20240503 02:13:08.522065 15354 consensus_queue.cc:260] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.522552 15365 ts_tablet_manager.cc:1909] T f9941d097fe94dd88a9abe7e3489a5bc P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:13:08.522845 15354 ts_tablet_manager.cc:1398] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20240503 02:13:08.523260 15048 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:13:08.524240 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table_new_name [id=a9303d04887745fdb936c571ea51f078]) successfully deleted
I20240503 02:13:08.534386 15330 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:08.534729 15330 ts_tablet_manager.cc:1367] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.156s user 0.005s sys 0.000s
I20240503 02:13:08.535816 15330 raft_consensus.cc:348] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.536247 15330 raft_consensus.cc:374] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.536367 15330 raft_consensus.cc:729] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.536761 15330 consensus_queue.cc:260] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.423429 15300 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.430184 10224 raft_consensus.cc:1261] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:08.436923 15332 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:08.437460 15349 tablet_copy_client.cc:308] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:08.539462 15301 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.541007 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet e5968b9989fb4f37bb4422645e7c8032 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:38795)
I20240503 02:13:08.541227 15332 ts_tablet_manager.cc:1367] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.391s user 0.004s sys 0.015s
I20240503 02:13:08.541329 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet e5968b9989fb4f37bb4422645e7c8032 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:38795: session id = eff8e8d11a4447218741bb994aa41082-e5968b9989fb4f37bb4422645e7c8032
I20240503 02:13:08.542429 15324 consensus_queue.cc:1035] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:08.542574 15332 raft_consensus.cc:348] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.542863 15332 raft_consensus.cc:374] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.544131 10491 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.544978 15332 raft_consensus.cc:729] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.545336 15332 consensus_queue.cc:260] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.546098 13061 tablet_copy_source_session.cc:205] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:08.546164 15332 ts_tablet_manager.cc:1398] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:08.547808 15298 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.003s
I20240503 02:13:08.548992 15349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e5968b9989fb4f37bb4422645e7c8032. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:13:08.557286 15373 log.cc:924] Time spent T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240503 02:13:08.570598 10224 raft_consensus.cc:2932] T 67be061441154fbba1e227d32474cd5e P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.575156 15349 tablet_copy_client.cc:783] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:08.575553 15349 tablet_copy_client.cc:647] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:08.576426 10207 tablet_service.cc:1467] Processing CreateTablet for tablet a4c033a336e845c9b5d5053d99bbf9e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5e54310acd2041b3a3cac4ad590fc045]), partition=
I20240503 02:13:08.576972 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c033a336e845c9b5d5053d99bbf9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.577313 15330 ts_tablet_manager.cc:1398] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.042s user 0.002s sys 0.000s
I20240503 02:13:08.577700 12886 tablet_service.cc:1467] Processing CreateTablet for tablet a4c033a336e845c9b5d5053d99bbf9e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5e54310acd2041b3a3cac4ad590fc045]), partition=
I20240503 02:13:08.578279 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c033a336e845c9b5d5053d99bbf9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.585320 15336 ts_tablet_manager.cc:1398] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.077s user 0.000s sys 0.002s
I20240503 02:13:08.586165 15332 tablet_bootstrap.cc:492] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:08.589112 15332 tablet_bootstrap.cc:654] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.596136 12907 raft_consensus.cc:1261] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.597204 13020 tablet_service.cc:1467] Processing CreateTablet for tablet a4c033a336e845c9b5d5053d99bbf9e8 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=5e54310acd2041b3a3cac4ad590fc045]), partition=
I20240503 02:13:08.597409 15349 tablet_copy_client.cc:523] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:08.597738 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a4c033a336e845c9b5d5053d99bbf9e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:08.599052 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-e3bba75394f24d4c91562089abad24c4 received from {username='slave'} at 127.4.5.65:40827
I20240503 02:13:08.599197 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-e3bba75394f24d4c91562089abad24c4 on tablet e3bba75394f24d4c91562089abad24c4 with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:08.599431 15270 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.601413 15349 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:08.608569 15393 ts_tablet_manager.cc:902] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:08.615106 14733 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:08.617586 15354 tablet_bootstrap.cc:492] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:08.621352 15048 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:08.623013 10489 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:08.630200 10357 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:08.633882 9950 catalog_manager.cc:5484] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:08.634613 9950 catalog_manager.cc:5484] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.626248 12905 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:08.643898 13305 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:08.645648 15063 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:08.646795 15271 raft_consensus.cc:482] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.647002 15271 raft_consensus.cc:504] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.653515 13040 consensus_queue.cc:237] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:08.654217 15271 leader_election.cc:288] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:08.657364 10357 raft_consensus.cc:1261] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:08.657964 10226 raft_consensus.cc:1261] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.658546 15392 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:08.659883 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cf65124d3a4ccd8eded963fb7e92e8" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:08.620256 15354 tablet_bootstrap.cc:654] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.660162 13040 raft_consensus.cc:2454] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:08.660629 13246 leader_election.cc:302] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:13:08.660979 15271 raft_consensus.cc:2781] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.661130 15271 raft_consensus.cc:482] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.661232 15271 raft_consensus.cc:3037] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.661506 10489 raft_consensus.cc:1261] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:08.661588 15301 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:08.664767 15271 raft_consensus.cc:504] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.665874 15271 leader_election.cc:288] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:08.666733 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cf65124d3a4ccd8eded963fb7e92e8" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:08.667066 13040 raft_consensus.cc:3037] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.668901 15392 raft_consensus.cc:482] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.669193 15392 raft_consensus.cc:504] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.670604 13040 raft_consensus.cc:2454] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:08.671123 15392 leader_election.cc:288] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:08.671398 15392 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:08.672049 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a129a73db84b74af353de0ec089c93" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:08.672381 10094 raft_consensus.cc:2379] T 44a129a73db84b74af353de0ec089c93 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8d454a3607d042dc9559ed59c3ccd84b in current term 1: Already voted for candidate f93b103af083471b8e8c0376e9148aad in this term.
I20240503 02:13:08.673403 13246 leader_election.cc:302] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:13:08.674492 10359 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:08.675397 15271 raft_consensus.cc:2781] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.675626 15271 raft_consensus.cc:686] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:08.676024 15271 consensus_queue.cc:237] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.676688 15270 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:08.681789 15336 tablet_bootstrap.cc:492] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:08.682699 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "44a129a73db84b74af353de0ec089c93" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:08.683478 12976 leader_election.cc:302] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b; no voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:08.683780 15336 tablet_bootstrap.cc:654] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:08.685251 13305 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:08.686550 15170 raft_consensus.cc:3037] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.688198 10488 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:08.689096 10356 raft_consensus.cc:1261] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:08.690147 12906 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:08.677543 9946 catalog_manager.cc:5484] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:08.692970 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cf65124d3a4ccd8eded963fb7e92e8" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:08.693369 12904 raft_consensus.cc:3037] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.693879 10491 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:08.695459 12905 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:08.699553 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8cf65124d3a4ccd8eded963fb7e92e8" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:08.699718 15349 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:08.700229 15349 tablet_bootstrap.cc:492] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:08.701714 12904 raft_consensus.cc:2454] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:08.707525 9947 catalog_manager.cc:5484] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:08.709582 10357 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:08.715514 9952 catalog_manager.cc:5484] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.715175 10224 consensus_queue.cc:237] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:08.720784 15349 ts_tablet_manager.cc:1367] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.119s user 0.000s sys 0.017s
I20240503 02:13:08.721940 15349 raft_consensus.cc:348] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:08.722375 15349 raft_consensus.cc:729] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:08.722786 15349 consensus_queue.cc:260] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:08.723587 12905 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:08.731473 15170 raft_consensus.cc:2726] T 44a129a73db84b74af353de0ec089c93 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:08.724422 15301 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20240503 02:13:08.743271 15380 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:08.746457 12904 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:08.754370 14793 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.756135 14792 raft_consensus.cc:1050] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:08.757051 14792 consensus_queue.cc:237] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:08.757632 15332 tablet_bootstrap.cc:492] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:08.757833 15332 ts_tablet_manager.cc:1367] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.172s user 0.008s sys 0.000s
I20240503 02:13:08.757794 9944 catalog_manager.cc:5484] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:08.758780 15332 raft_consensus.cc:348] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.759105 15332 raft_consensus.cc:374] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.759280 15332 raft_consensus.cc:729] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.761094 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af7beeafa08f477796e5070fbc177dcc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:08.761654 15332 consensus_queue.cc:260] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.762848 15332 ts_tablet_manager.cc:1398] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:08.766569 10358 consensus_queue.cc:237] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:08.770344 13305 raft_consensus.cc:1261] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:08.771194 15063 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:08.772056 10490 raft_consensus.cc:1261] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:08.774550 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet af7beeafa08f477796e5070fbc177dcc with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:08.778309 10094 raft_consensus.cc:1203] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:08.785920 15345 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:08.787545 15351 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:08.790418 10489 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:08.766173 15350 ts_tablet_manager.cc:1888] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:08.791358 15350 log.cc:1196] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/f9941d097fe94dd88a9abe7e3489a5bc
I20240503 02:13:08.762168 15349 ts_tablet_manager.cc:1398] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.041s user 0.000s sys 0.002s
I20240503 02:13:08.792037 15350 ts_tablet_manager.cc:1909] T f9941d097fe94dd88a9abe7e3489a5bc P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:13:08.791988 10490 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:08.793660 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet f9941d097fe94dd88a9abe7e3489a5bc (table rebalancer_dynamic_table_new_name [id=a9303d04887745fdb936c571ea51f078]) successfully deleted
I20240503 02:13:08.795089 15301 raft_consensus.cc:482] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.795279 15301 raft_consensus.cc:504] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.795799 13305 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:08.796697 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c014b2069ae42788aae87d8304fa155" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:08.796898 15301 leader_election.cc:288] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:08.796950 13302 raft_consensus.cc:2454] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:08.797555 10159 leader_election.cc:302] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:08.798257 15382 ts_tablet_manager.cc:1888] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:08.799152 15380 raft_consensus.cc:2781] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.799206 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-e5968b9989fb4f37bb4422645e7c8032 received from {username='slave'} at 127.4.5.66:38795
I20240503 02:13:08.799309 15380 raft_consensus.cc:482] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.799391 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session eff8e8d11a4447218741bb994aa41082-e5968b9989fb4f37bb4422645e7c8032 on tablet e5968b9989fb4f37bb4422645e7c8032 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:08.799423 15380 raft_consensus.cc:3037] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.800894 15404 tablet_replica.cc:330] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:08.803386 15404 raft_consensus.cc:2229] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.803664 15380 raft_consensus.cc:504] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.804868 15380 leader_election.cc:288] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:08.805670 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c014b2069ae42788aae87d8304fa155" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:08.806000 13302 raft_consensus.cc:3037] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.807013 15404 raft_consensus.cc:2258] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.808996 15404 ts_tablet_manager.cc:1875] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:08.809924 9947 catalog_manager.cc:5484] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 5 to 6, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:08.810335 15405 raft_consensus.cc:482] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.810573 15382 log.cc:1196] T 67be061441154fbba1e227d32474cd5e P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/67be061441154fbba1e227d32474cd5e
I20240503 02:13:08.810519 15405 raft_consensus.cc:504] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.811295 15405 leader_election.cc:288] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:08.813987 13302 raft_consensus.cc:2454] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:08.814476 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 67be061441154fbba1e227d32474cd5e (table rebalancer_extra_table_2 [id=79529b4d17da4f71a9dad3d301ddd43d]) successfully deleted
I20240503 02:13:08.814599 10159 leader_election.cc:302] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:08.815104 15301 raft_consensus.cc:2781] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.815066 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c014b2069ae42788aae87d8304fa155" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:08.815405 13302 raft_consensus.cc:2379] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 031a0afb01b6469299fa829fa3467ef7 in current term 1: Already voted for candidate eff8e8d11a4447218741bb994aa41082 in this term.
I20240503 02:13:08.815560 15301 raft_consensus.cc:686] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:08.815969 15301 consensus_queue.cc:237] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:08.821424 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet 5c762709f66f44b98089e69cbace1570 with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:36690
W20240503 02:13:08.826848 10028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 71217.7us
I20240503 02:13:08.828776 10091 raft_consensus.cc:2932] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:08.840061 15407 tablet_replica.cc:330] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:08.840341 15407 raft_consensus.cc:2229] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.840704 15407 raft_consensus.cc:2258] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.849289 15354 tablet_bootstrap.cc:492] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:08.849763 15354 ts_tablet_manager.cc:1367] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.232s user 0.004s sys 0.032s
I20240503 02:13:08.851065 15354 raft_consensus.cc:348] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.851460 15354 raft_consensus.cc:374] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.851617 15354 raft_consensus.cc:729] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.851984 15354 consensus_queue.cc:260] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.852804 15354 ts_tablet_manager.cc:1398] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:08.853363 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 87bb6c6f9ea646a5a26106312a3742eb with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:08.856598 9943 catalog_manager.cc:5484] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:08.857080 15404 ts_tablet_manager.cc:1888] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:08.858682 15405 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:08.861269 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c014b2069ae42788aae87d8304fa155" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:08.861662 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c014b2069ae42788aae87d8304fa155" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:08.861919 10094 raft_consensus.cc:2454] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:08.860131 9951 catalog_manager.cc:5484] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:08.862061 10358 raft_consensus.cc:1261] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.862398 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c014b2069ae42788aae87d8304fa155" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:08.862571 10094 raft_consensus.cc:3037] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.863601 13305 raft_consensus.cc:1261] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.864377 15405 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.865682 10094 raft_consensus.cc:2454] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:08.867074 15407 ts_tablet_manager.cc:1875] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:08.880262 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5c762709f66f44b98089e69cbace1570 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:08.884191 10355 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:08.884912 15404 log.cc:1196] T af7beeafa08f477796e5070fbc177dcc P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/af7beeafa08f477796e5070fbc177dcc
I20240503 02:13:08.885493 15380 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:08.887080 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet af7beeafa08f477796e5070fbc177dcc (table rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]) successfully deleted
I20240503 02:13:08.888404 10028 leader_election.cc:302] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082
I20240503 02:13:08.888675 10355 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:08.889845 15405 raft_consensus.cc:2726] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:08.877321 12886 tablet_service.cc:1514] Processing DeleteTablet for tablet 87bb6c6f9ea646a5a26106312a3742eb with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:08.896524 15405 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.878824 12908 raft_consensus.cc:1261] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:08.902307 15405 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:08.904274 15336 tablet_bootstrap.cc:492] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:08.904541 15336 ts_tablet_manager.cc:1367] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.223s user 0.009s sys 0.022s
I20240503 02:13:08.905560 15336 raft_consensus.cc:348] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.906847 15336 raft_consensus.cc:374] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:08.907116 15336 raft_consensus.cc:729] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:08.908521 15336 consensus_queue.cc:260] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.910355 15380 raft_consensus.cc:482] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:08.910573 15380 raft_consensus.cc:504] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.911458 15380 leader_election.cc:288] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:08.913368 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c033a336e845c9b5d5053d99bbf9e8" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:08.913607 13038 raft_consensus.cc:2454] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:08.912302 15231 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.914054 10162 leader_election.cc:302] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:08.914376 15380 raft_consensus.cc:2781] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:08.914505 15380 raft_consensus.cc:482] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:08.914631 15380 raft_consensus.cc:3037] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.914790 15336 ts_tablet_manager.cc:1398] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20240503 02:13:08.916250 12905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c033a336e845c9b5d5053d99bbf9e8" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:08.917479 12905 raft_consensus.cc:2454] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:08.917740 15380 raft_consensus.cc:504] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.918844 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c033a336e845c9b5d5053d99bbf9e8" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:08.919087 13038 raft_consensus.cc:3037] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.919333 15380 leader_election.cc:288] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:08.919849 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a4c033a336e845c9b5d5053d99bbf9e8" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:08.920092 12904 raft_consensus.cc:3037] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:08.920048 14793 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.923373 13038 raft_consensus.cc:2454] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:08.924053 12904 raft_consensus.cc:2454] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:08.924583 10159 leader_election.cc:302] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:08.925154 15380 raft_consensus.cc:2781] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:08.925365 15380 raft_consensus.cc:686] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:08.925848 15380 consensus_queue.cc:237] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:08.925974 12907 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.935518 15414 tablet_replica.cc:330] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:08.935823 15414 raft_consensus.cc:2229] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:08.936131 15414 raft_consensus.cc:2258] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:08.937840 15240 consensus_queue.cc:1035] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.939808 10354 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.948568 9946 catalog_manager.cc:5484] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.953984 13040 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.954614 15407 ts_tablet_manager.cc:1888] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:08.954746 15407 log.cc:1196] T 5c762709f66f44b98089e69cbace1570 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/5c762709f66f44b98089e69cbace1570
I20240503 02:13:08.960310 9943 catalog_manager.cc:5484] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.968398 13305 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:08.979362 15209 consensus_queue.cc:1035] T 44a129a73db84b74af353de0ec089c93 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:08.987296 10224 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:08.989234 10091 consensus_queue.cc:237] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:08.993203 15414 ts_tablet_manager.cc:1875] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:08.996768 13040 raft_consensus.cc:1261] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:08.996896 10359 raft_consensus.cc:1261] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:08.997622 15405 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:08.998195 15420 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:09.001583 15405 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:09.006415 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 476f6e414dae4db3ba9a187adad75f8c with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:09.007416 13040 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:09.016517 10357 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:09.018838 9946 catalog_manager.cc:5484] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:09.039187 13302 consensus_queue.cc:237] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:09.025660 10354 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:09.052467 13174 raft_consensus.cc:1261] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:09.053159 15271 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:09.053479 13245 consensus_peers.cc:485] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 83f291bfdc064894ae0f7d6fc58655df. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:09.055874 13039 raft_consensus.cc:1261] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:09.057730 15271 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:09.073904 15431 ts_tablet_manager.cc:902] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:09.074872 15393 tablet_copy_client.cc:308] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:13:09.076599 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet 31f72bc850c74289aa3a59ecc3dcc287 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:41551)
I20240503 02:13:09.076851 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet 31f72bc850c74289aa3a59ecc3dcc287 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:41551: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-31f72bc850c74289aa3a59ecc3dcc287
I20240503 02:13:09.077044 15369 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:09.079907 10511 tablet_copy_source_session.cc:205] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:09.082324 9947 catalog_manager.cc:5484] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:09.094601 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 5c762709f66f44b98089e69cbace1570 (table rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]) successfully deleted
I20240503 02:13:09.099234 15414 ts_tablet_manager.cc:1888] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:09.103283 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 476f6e414dae4db3ba9a187adad75f8c with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:09.103883 15350 tablet_replica.cc:330] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:09.104236 15350 raft_consensus.cc:2229] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:09.104568 15350 raft_consensus.cc:2258] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:09.108783 15350 ts_tablet_manager.cc:1875] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:09.112598 10094 consensus_queue.cc:237] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:09.077881 13172 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:09.117823 10357 raft_consensus.cc:1261] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:09.118981 15350 ts_tablet_manager.cc:1888] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:09.118734 9946 catalog_manager.cc:5484] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:09.117007 15393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 31f72bc850c74289aa3a59ecc3dcc287. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.117208 12904 raft_consensus.cc:1261] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:09.121444 15418 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:09.122383 15231 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:09.124310 13039 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:09.125584 15414 log.cc:1196] T 87bb6c6f9ea646a5a26106312a3742eb P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/87bb6c6f9ea646a5a26106312a3742eb
I20240503 02:13:09.126637 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 87bb6c6f9ea646a5a26106312a3742eb (table rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]) successfully deleted
I20240503 02:13:09.131438 15231 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:09.141075 12908 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:09.144594 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 46e5690a42d44500b3e6ca24d771affc with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:09.147789 10354 raft_consensus.cc:2932] T 46e5690a42d44500b3e6ca24d771affc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:09.148527 15350 log.cc:1196] T 476f6e414dae4db3ba9a187adad75f8c P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/476f6e414dae4db3ba9a187adad75f8c
I20240503 02:13:09.151176 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 476f6e414dae4db3ba9a187adad75f8c (table rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]) successfully deleted
I20240503 02:13:09.156155 9946 catalog_manager.cc:5484] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:09.163067 10226 raft_consensus.cc:1203] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:09.187987 15393 tablet_copy_client.cc:783] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:09.188431 15393 tablet_copy_client.cc:647] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:09.192288 15393 tablet_copy_client.cc:523] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:09.197093 15271 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.208067 15431 tablet_copy_client.cc:308] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:13:09.210491 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet dd081871c430470b95fce8f19634dc2e from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:33045)
I20240503 02:13:09.210803 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet dd081871c430470b95fce8f19634dc2e from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:33045: session id = f93b103af083471b8e8c0376e9148aad-dd081871c430470b95fce8f19634dc2e
I20240503 02:13:09.220008 10511 tablet_copy_source_session.cc:205] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:09.222149 15431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd081871c430470b95fce8f19634dc2e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.236707 9944 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:09.238574 15431 tablet_copy_client.cc:783] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:09.238997 15431 tablet_copy_client.cc:647] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:09.248888 15431 tablet_copy_client.cc:523] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:09.254603 15431 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:09.208068 15393 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:09.262915 15369 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.265043 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 46e5690a42d44500b3e6ca24d771affc with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:09.269235 12904 raft_consensus.cc:1261] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:09.272042 15301 consensus_queue.cc:1035] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.282464 15301 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.285065 13039 raft_consensus.cc:1261] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:09.285874 15301 consensus_queue.cc:1035] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.290100 15452 tablet_replica.cc:330] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:09.291318 15452 raft_consensus.cc:2229] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:09.291831 15452 raft_consensus.cc:2258] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:09.297829 15452 ts_tablet_manager.cc:1875] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:09.323937 15431 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:09.326217 15466 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:09.327025 15466 tablet.cc:1620] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:09.340591 15465 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:09.341233 15431 tablet_bootstrap.cc:492] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:09.341634 15431 ts_tablet_manager.cc:1367] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.087s user 0.023s sys 0.009s
I20240503 02:13:09.342654 15431 raft_consensus.cc:348] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:09.342955 15431 raft_consensus.cc:729] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:09.343233 15431 consensus_queue.cc:260] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:09.343626 15452 ts_tablet_manager.cc:1888] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:09.343822 15452 log.cc:1196] T 46e5690a42d44500b3e6ca24d771affc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/46e5690a42d44500b3e6ca24d771affc
I20240503 02:13:09.349642 15431 ts_tablet_manager.cc:1398] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:13:09.350482 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-dd081871c430470b95fce8f19634dc2e received from {username='slave'} at 127.4.5.71:33045
I20240503 02:13:09.350656 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session f93b103af083471b8e8c0376e9148aad-dd081871c430470b95fce8f19634dc2e on tablet dd081871c430470b95fce8f19634dc2e with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:09.350584 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 46e5690a42d44500b3e6ca24d771affc (table rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]) successfully deleted
I20240503 02:13:09.354435 9947 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet a4c033a336e845c9b5d5053d99bbf9e8 (table rebalancer_dynamic_table [id=5e54310acd2041b3a3cac4ad590fc045]). Expected version 1 got 0
I20240503 02:13:09.356043 15468 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:09.357718 15300 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:09.355175 15465 tablet.cc:1620] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:09.359347 15468 tablet.cc:1620] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:09.365491 9946 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet a4c033a336e845c9b5d5053d99bbf9e8 (table rebalancer_dynamic_table [id=5e54310acd2041b3a3cac4ad590fc045]). Expected version 1 got 0
I20240503 02:13:09.377187 15399 raft_consensus.cc:1050] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:09.378367 15399 consensus_queue.cc:237] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:09.381078 9944 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=5e54310acd2041b3a3cac4ad590fc045] alter complete (version 1)
I20240503 02:13:09.397593 13172 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.399477 10091 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.400597 15270 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.401394 15270 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.411532 13305 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.417234 15399 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.451670 15393 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:09.452363 15393 tablet_bootstrap.cc:492] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:09.452854 15393 ts_tablet_manager.cc:1367] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.245s user 0.068s sys 0.006s
I20240503 02:13:09.454032 15393 raft_consensus.cc:348] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:09.454429 15393 raft_consensus.cc:729] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:09.454725 15393 consensus_queue.cc:260] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:09.458189 15392 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:09.462419 15393 ts_tablet_manager.cc:1398] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 02:13:09.466075 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-31f72bc850c74289aa3a59ecc3dcc287 received from {username='slave'} at 127.4.5.72:41551
I20240503 02:13:09.466297 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-31f72bc850c74289aa3a59ecc3dcc287 on tablet 31f72bc850c74289aa3a59ecc3dcc287 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:09.469681 13305 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:09.476970 9951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:09.482828 9947 catalog_manager.cc:5484] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:09.486084 13174 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:09.488674 13168 raft_consensus.cc:1203] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:09.492344 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:09.513751 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 16f89ddc03524fa2bb6e5c66211564c0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.514379 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16f89ddc03524fa2bb6e5c66211564c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.516216 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 16f89ddc03524fa2bb6e5c66211564c0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.516830 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16f89ddc03524fa2bb6e5c66211564c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.518002 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 0df3294fa2d74baea8c573a5dcfa96bd (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.518467 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df3294fa2d74baea8c573a5dcfa96bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.521307 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 16f89ddc03524fa2bb6e5c66211564c0 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.521936 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16f89ddc03524fa2bb6e5c66211564c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.525313 15465 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:09.526001 15465 tablet.cc:1620] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:09.526202 15468 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:09.526755 15468 tablet.cc:1620] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:09.546864 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 0df3294fa2d74baea8c573a5dcfa96bd (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.547596 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df3294fa2d74baea8c573a5dcfa96bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.548434 13153 tablet_service.cc:1467] Processing CreateTablet for tablet af66d6e994ab4049964f4218e05ef886 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.549041 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af66d6e994ab4049964f4218e05ef886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.553822 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 0df3294fa2d74baea8c573a5dcfa96bd (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.554399 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df3294fa2d74baea8c573a5dcfa96bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.558881 15466 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:09.559558 15466 tablet.cc:1620] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:09.564774 13289 tablet_service.cc:1467] Processing CreateTablet for tablet af66d6e994ab4049964f4218e05ef886 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.565394 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af66d6e994ab4049964f4218e05ef886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.567647 10091 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:09.569556 13019 tablet_service.cc:1467] Processing CreateTablet for tablet af66d6e994ab4049964f4218e05ef886 (DEFAULT_TABLE table=rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:09.570170 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af66d6e994ab4049964f4218e05ef886. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.585892 15474 raft_consensus.cc:1050] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:09.586993 15474 consensus_queue.cc:237] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:09.591207 15479 tablet_bootstrap.cc:492] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:09.593639 15479 tablet_bootstrap.cc:654] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.603147 15483 tablet_bootstrap.cc:492] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:09.605809 15482 tablet_bootstrap.cc:492] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:09.608054 15481 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:09.612247 10224 raft_consensus.cc:1261] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.612419 12904 raft_consensus.cc:1261] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.612977 15487 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:09.613813 13172 raft_consensus.cc:1261] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.615860 15487 tablet_bootstrap.cc:654] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.621111 15479 tablet_bootstrap.cc:492] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:09.621419 15479 ts_tablet_manager.cc:1367] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.035s user 0.001s sys 0.008s
I20240503 02:13:09.622555 15479 raft_consensus.cc:348] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.622964 15479 raft_consensus.cc:374] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.623253 15479 raft_consensus.cc:729] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.623653 15479 consensus_queue.cc:260] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.627974 15479 ts_tablet_manager.cc:1398] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:09.632339 15474 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.633144 15474 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.637943 15248 raft_consensus.cc:482] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.638162 15248 raft_consensus.cc:504] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.639283 15248 leader_election.cc:288] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:09.640411 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f89ddc03524fa2bb6e5c66211564c0" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
W20240503 02:13:09.641201 10427 leader_election.cc:341] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:09.642083 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f89ddc03524fa2bb6e5c66211564c0" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
W20240503 02:13:09.643051 10427 leader_election.cc:341] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:09.643241 10427 leader_election.cc:302] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ec2f1283994a3089741715f811c850; no voters: 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:09.643913 15248 raft_consensus.cc:2726] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:09.654750 15482 tablet_bootstrap.cc:654] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.655812 15481 tablet_bootstrap.cc:654] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.657480 15483 tablet_bootstrap.cc:654] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.659175 15480 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:09.661453 9947 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=5e54310acd2041b3a3cac4ad590fc045] alter complete (version 2)
I20240503 02:13:09.676139 15480 tablet_bootstrap.cc:654] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.678750 15492 ts_tablet_manager.cc:902] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:09.678957 13305 raft_consensus.cc:1203] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:09.687675 15483 tablet_bootstrap.cc:492] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:09.688023 15483 ts_tablet_manager.cc:1367] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.085s user 0.003s sys 0.004s
I20240503 02:13:09.689556 15483 raft_consensus.cc:348] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.689905 15483 raft_consensus.cc:374] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.690059 15483 raft_consensus.cc:729] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.690552 15483 consensus_queue.cc:260] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.692595 15392 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:09.697219 15483 ts_tablet_manager.cc:1398] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:09.698899 15481 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:09.699153 15481 ts_tablet_manager.cc:1367] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.091s user 0.005s sys 0.000s
I20240503 02:13:09.701797 15481 raft_consensus.cc:348] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.703869 15481 raft_consensus.cc:374] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.704054 15481 raft_consensus.cc:729] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.704456 15481 consensus_queue.cc:260] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.707859 15480 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:09.708173 15480 ts_tablet_manager.cc:1367] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.003s
I20240503 02:13:09.709439 15480 raft_consensus.cc:348] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.709784 15480 raft_consensus.cc:374] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.709954 15480 raft_consensus.cc:729] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.710289 15480 consensus_queue.cc:260] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.712313 15480 ts_tablet_manager.cc:1398] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:09.723508 12904 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:09.723799 10224 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:09.733022 9943 catalog_manager.cc:5484] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:09.737357 15482 tablet_bootstrap.cc:492] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:09.737794 15482 ts_tablet_manager.cc:1367] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.003s
I20240503 02:13:09.739339 15482 raft_consensus.cc:348] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.739586 15482 raft_consensus.cc:374] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.739756 15482 raft_consensus.cc:729] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.740053 15482 consensus_queue.cc:260] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.745447 15487 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:09.745653 15487 ts_tablet_manager.cc:1367] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.133s user 0.042s sys 0.059s
I20240503 02:13:09.747193 15487 raft_consensus.cc:348] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.747488 15487 raft_consensus.cc:374] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.747604 15487 raft_consensus.cc:729] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.747941 15487 consensus_queue.cc:260] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.749069 15487 ts_tablet_manager.cc:1398] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:09.721477 15481 ts_tablet_manager.cc:1398] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20240503 02:13:09.746776 15392 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.756278 15483 tablet_bootstrap.cc:492] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:09.759276 13039 consensus_queue.cc:237] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:09.760551 15480 tablet_bootstrap.cc:492] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:09.762248 15482 ts_tablet_manager.cc:1398] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20240503 02:13:09.762656 15482 tablet_bootstrap.cc:492] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:09.766407 10224 raft_consensus.cc:1261] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:09.766574 15483 tablet_bootstrap.cc:654] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.769567 9950 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:09.774003 15480 tablet_bootstrap.cc:654] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.776485 15482 tablet_bootstrap.cc:654] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:09.777137 15392 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:09.780393 13172 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:09.797850 12908 raft_consensus.cc:1261] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:09.798614 15480 tablet_bootstrap.cc:492] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:09.798851 15480 ts_tablet_manager.cc:1367] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.038s user 0.005s sys 0.003s
I20240503 02:13:09.799157 15483 tablet_bootstrap.cc:492] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:09.799371 15483 ts_tablet_manager.cc:1367] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.005s
I20240503 02:13:09.799696 15480 raft_consensus.cc:348] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.800223 15480 raft_consensus.cc:374] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.800359 15480 raft_consensus.cc:729] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.800282 15483 raft_consensus.cc:348] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.800753 15480 consensus_queue.cc:260] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.803014 15480 ts_tablet_manager.cc:1398] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:09.802954 10224 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:09.800434 15392 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:09.809058 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e5968b9989fb4f37bb4422645e7c8032 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:09.810201 15483 raft_consensus.cc:374] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.810343 15483 raft_consensus.cc:729] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.811453 15483 consensus_queue.cc:260] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.814340 15483 ts_tablet_manager.cc:1398] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20240503 02:13:09.811770 9943 catalog_manager.cc:5484] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:09.820515 15392 consensus_queue.cc:1035] T e5968b9989fb4f37bb4422645e7c8032 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:09.840184 15465 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:09.841039 15298 raft_consensus.cc:482] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.841840 15298 raft_consensus.cc:504] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.843842 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df3294fa2d74baea8c573a5dcfa96bd" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:09.844123 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df3294fa2d74baea8c573a5dcfa96bd" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:09.844225 13302 raft_consensus.cc:2454] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:09.844401 12907 raft_consensus.cc:2454] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:09.850121 15466 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:09.842667 15298 leader_election.cc:288] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:09.856571 12908 raft_consensus.cc:2932] T e5968b9989fb4f37bb4422645e7c8032 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:09.857419 10159 leader_election.cc:302] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:09.858286 15378 raft_consensus.cc:2781] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.858397 15378 raft_consensus.cc:482] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.858500 15378 raft_consensus.cc:3037] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.858520 13152 tablet_service.cc:1514] Processing DeleteTablet for tablet e5968b9989fb4f37bb4422645e7c8032 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:09.862478 15482 tablet_bootstrap.cc:492] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:09.862927 15438 raft_consensus.cc:482] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.863154 15438 raft_consensus.cc:504] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.864084 15482 ts_tablet_manager.cc:1367] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.005s
I20240503 02:13:09.864130 15438 leader_election.cc:288] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:09.865613 15482 raft_consensus.cc:348] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.866092 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af66d6e994ab4049964f4218e05ef886" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:09.866415 13040 raft_consensus.cc:2454] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:09.868388 15482 raft_consensus.cc:374] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:09.868546 15482 raft_consensus.cc:729] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:09.870134 15482 consensus_queue.cc:260] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.870965 15482 ts_tablet_manager.cc:1398] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.007s user 0.000s sys 0.003s
I20240503 02:13:09.871007 13246 leader_election.cc:302] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:13:09.871490 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af66d6e994ab4049964f4218e05ef886" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:09.871730 13174 raft_consensus.cc:2454] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:09.872617 15240 raft_consensus.cc:482] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:09.872764 15240 raft_consensus.cc:504] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.873661 15240 leader_election.cc:288] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:09.874402 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f89ddc03524fa2bb6e5c66211564c0" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:09.874698 10490 raft_consensus.cc:2454] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:09.875339 13107 leader_election.cc:302] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:09.875605 15438 raft_consensus.cc:2781] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.875757 15438 raft_consensus.cc:482] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.875854 15438 raft_consensus.cc:3037] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.875957 15240 raft_consensus.cc:2781] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:09.876078 15240 raft_consensus.cc:482] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:09.876178 15240 raft_consensus.cc:3037] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.879040 15240 raft_consensus.cc:504] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.879040 15378 raft_consensus.cc:504] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.879884 15378 leader_election.cc:288] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:09.879949 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f89ddc03524fa2bb6e5c66211564c0" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:09.880265 13040 raft_consensus.cc:2454] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:09.880777 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f89ddc03524fa2bb6e5c66211564c0" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:09.881031 10491 raft_consensus.cc:3037] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.887486 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df3294fa2d74baea8c573a5dcfa96bd" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:09.887835 12904 raft_consensus.cc:3037] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.891060 12904 raft_consensus.cc:2454] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:09.892764 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0df3294fa2d74baea8c573a5dcfa96bd" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:09.893105 13305 raft_consensus.cc:3037] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.896553 10491 raft_consensus.cc:2454] T 16f89ddc03524fa2bb6e5c66211564c0 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:09.897291 13107 leader_election.cc:302] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:09.897696 15059 raft_consensus.cc:2781] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.898272 15468 tablet.cc:1722] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:09.898301 10159 leader_election.cc:302] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:09.901504 15240 leader_election.cc:288] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:09.903517 13305 raft_consensus.cc:2454] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:09.907272 15298 raft_consensus.cc:2781] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.907498 15298 raft_consensus.cc:686] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:09.907855 15298 consensus_queue.cc:237] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:09.912181 15492 tablet_copy_client.cc:308] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.72:33671
I20240503 02:13:09.913604 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16f89ddc03524fa2bb6e5c66211564c0" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:09.913966 13040 raft_consensus.cc:3037] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.916172 13330 tablet_copy_service.cc:140] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Received BeginTabletCopySession request for tablet 83f291bfdc064894ae0f7d6fc58655df from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:56309)
I20240503 02:13:09.916420 13330 tablet_copy_service.cc:161] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Beginning new tablet copy session on tablet 83f291bfdc064894ae0f7d6fc58655df from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:56309: session id = eff8e8d11a4447218741bb994aa41082-83f291bfdc064894ae0f7d6fc58655df
I20240503 02:13:09.918079 15508 tablet_replica.cc:330] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:09.919822 13330 tablet_copy_source_session.cc:205] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:09.920279 15508 raft_consensus.cc:2229] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:09.920882 15508 raft_consensus.cc:2258] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:09.922734 9943 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=5e54310acd2041b3a3cac4ad590fc045] alter complete (version 3)
I20240503 02:13:09.932739 15438 raft_consensus.cc:504] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.936964 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af66d6e994ab4049964f4218e05ef886" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:09.937265 13174 raft_consensus.cc:3037] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.939532 15492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 83f291bfdc064894ae0f7d6fc58655df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:09.946723 15496 raft_consensus.cc:1050] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:09.948261 15496 consensus_queue.cc:237] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:09.949458 15508 ts_tablet_manager.cc:1875] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:09.950632 13040 raft_consensus.cc:2454] T 16f89ddc03524fa2bb6e5c66211564c0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:09.951813 13174 raft_consensus.cc:2454] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:09.952730 13246 leader_election.cc:302] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:09.953282 15437 raft_consensus.cc:2781] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:09.961292 9946 catalog_manager.cc:5484] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:09.962162 15438 leader_election.cc:288] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:09.962447 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "af66d6e994ab4049964f4218e05ef886" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:09.962843 13039 raft_consensus.cc:3037] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:09.962898 15059 raft_consensus.cc:686] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:13:09.964138 12908 raft_consensus.cc:1261] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.965312 13302 raft_consensus.cc:1261] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.965838 15248 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.966284 15437 raft_consensus.cc:686] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:09.966523 15446 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.966650 15437 consensus_queue.cc:237] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.967283 13039 raft_consensus.cc:2454] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:09.972105 13174 raft_consensus.cc:1261] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:09.973498 15446 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:09.963294 15059 consensus_queue.cc:237] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:09.987339 15496 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:09.990711 12904 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:10.022436 15492 tablet_copy_client.cc:783] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:10.022933 15492 tablet_copy_client.cc:647] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:10.023296 9943 catalog_manager.cc:5484] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:10.027957 9950 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:10.028276 9950 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:10.033308 9950 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet a4c033a336e845c9b5d5053d99bbf9e8
I20240503 02:13:10.034404 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c033a336e845c9b5d5053d99bbf9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:10 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:10.034539 13039 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4"
dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:35422
I20240503 02:13:10.034829 13039 raft_consensus.cc:595] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:10.035971 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c033a336e845c9b5d5053d99bbf9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:10 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:10.037358 12886 tablet_service.cc:1514] Processing DeleteTablet for tablet a4c033a336e845c9b5d5053d99bbf9e8 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:10 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:10.037814 15492 tablet_copy_client.cc:523] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:10.043915 15524 tablet_replica.cc:330] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:10.044816 15526 tablet_replica.cc:330] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:10.045392 15524 raft_consensus.cc:2229] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:10.045858 15526 raft_consensus.cc:2229] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:10.047354 15524 raft_consensus.cc:2258] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.047777 15526 raft_consensus.cc:2258] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.049221 15524 ts_tablet_manager.cc:1875] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:10.034170 15508 ts_tablet_manager.cc:1888] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:10.053750 15508 log.cc:1196] T e5968b9989fb4f37bb4422645e7c8032 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/e5968b9989fb4f37bb4422645e7c8032
I20240503 02:13:10.054572 15525 tablet_replica.cc:330] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:10.050407 15526 ts_tablet_manager.cc:1875] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:10.055081 15525 raft_consensus.cc:2229] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:10.055532 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet e5968b9989fb4f37bb4422645e7c8032 (table rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]) successfully deleted
I20240503 02:13:10.056968 9950 catalog_manager.cc:5484] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:10.065668 15492 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:10.070113 13174 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:10.087208 13305 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:10.096237 9946 catalog_manager.cc:5484] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 1, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:10.101688 15526 ts_tablet_manager.cc:1888] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:10.104287 10491 consensus_queue.cc:237] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:10.107715 15525 raft_consensus.cc:2258] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.108009 13174 raft_consensus.cc:1261] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:10.108650 15248 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:10.109299 15525 ts_tablet_manager.cc:1875] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:10.109814 15526 log.cc:1196] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/a4c033a336e845c9b5d5053d99bbf9e8
I20240503 02:13:10.110682 15524 ts_tablet_manager.cc:1888] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:10.112581 13302 raft_consensus.cc:1261] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:10.113438 15446 consensus_queue.cc:1035] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:10.113329 15496 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:10.115654 13168 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:10.115945 13305 raft_consensus.cc:2932] T dd081871c430470b95fce8f19634dc2e P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:10.121284 15526 ts_tablet_manager.cc:1909] T a4c033a336e845c9b5d5053d99bbf9e8 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:13:10.121588 15524 log.cc:1196] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/a4c033a336e845c9b5d5053d99bbf9e8
I20240503 02:13:10.121626 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet dd081871c430470b95fce8f19634dc2e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:10.123989 15524 ts_tablet_manager.cc:1909] T a4c033a336e845c9b5d5053d99bbf9e8 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:10.125512 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet a4c033a336e845c9b5d5053d99bbf9e8 (table rebalancer_dynamic_table_new_name [id=5e54310acd2041b3a3cac4ad590fc045]) successfully deleted
I20240503 02:13:10.126297 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet a4c033a336e845c9b5d5053d99bbf9e8 (table rebalancer_dynamic_table_new_name [id=5e54310acd2041b3a3cac4ad590fc045]) successfully deleted
I20240503 02:13:10.126803 9943 catalog_manager.cc:5484] T dd081871c430470b95fce8f19634dc2e P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:10.134819 12886 tablet_service.cc:1514] Processing DeleteTablet for tablet dd081871c430470b95fce8f19634dc2e with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:10.135982 15526 tablet_replica.cc:330] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:10.136353 15526 raft_consensus.cc:2229] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:10.136696 15526 raft_consensus.cc:2258] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.139652 9943 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:10.141012 15526 ts_tablet_manager.cc:1875] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:10.151039 15492 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:10.151791 15492 tablet_bootstrap.cc:492] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:10.152177 15492 ts_tablet_manager.cc:1367] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.087s user 0.006s sys 0.006s
I20240503 02:13:10.153674 15492 raft_consensus.cc:348] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:10.154002 15492 raft_consensus.cc:729] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:10.154248 15492 consensus_queue.cc:260] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:10.155035 15492 ts_tablet_manager.cc:1398] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20240503 02:13:10.156186 15525 ts_tablet_manager.cc:1888] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:10.156380 15525 log.cc:1196] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/a4c033a336e845c9b5d5053d99bbf9e8
I20240503 02:13:10.156390 13330 tablet_copy_service.cc:342] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-83f291bfdc064894ae0f7d6fc58655df received from {username='slave'} at 127.4.5.66:56309
I20240503 02:13:10.156600 13330 tablet_copy_service.cc:433] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: ending tablet copy session eff8e8d11a4447218741bb994aa41082-83f291bfdc064894ae0f7d6fc58655df on tablet 83f291bfdc064894ae0f7d6fc58655df with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:10.156950 15525 ts_tablet_manager.cc:1909] T a4c033a336e845c9b5d5053d99bbf9e8 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:13:10.158236 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet a4c033a336e845c9b5d5053d99bbf9e8 (table rebalancer_dynamic_table_new_name [id=5e54310acd2041b3a3cac4ad590fc045]) successfully deleted
I20240503 02:13:10.161355 15526 ts_tablet_manager.cc:1888] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:10.161720 15526 log.cc:1196] T dd081871c430470b95fce8f19634dc2e P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/dd081871c430470b95fce8f19634dc2e
I20240503 02:13:10.163640 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet dd081871c430470b95fce8f19634dc2e (table rebalancer_extra_table_8 [id=1b2aaa8785944255b7cf99640dc67368]) successfully deleted
I20240503 02:13:10.168663 13152 tablet_service.cc:1467] Processing CreateTablet for tablet bde4f280f95e467f98f907b02adfaaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e]), partition=
I20240503 02:13:10.168759 13020 tablet_service.cc:1467] Processing CreateTablet for tablet bde4f280f95e467f98f907b02adfaaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e]), partition=
I20240503 02:13:10.169129 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4f280f95e467f98f907b02adfaaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.169175 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4f280f95e467f98f907b02adfaaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.173642 12886 tablet_service.cc:1467] Processing CreateTablet for tablet bde4f280f95e467f98f907b02adfaaf0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e]), partition=
I20240503 02:13:10.174105 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bde4f280f95e467f98f907b02adfaaf0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.185390 15483 tablet_bootstrap.cc:492] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:10.187165 15483 tablet_bootstrap.cc:654] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.188784 15482 tablet_bootstrap.cc:492] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:10.189800 15481 tablet_bootstrap.cc:492] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:10.190446 15482 tablet_bootstrap.cc:654] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.191105 15481 tablet_bootstrap.cc:654] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.196844 15481 tablet_bootstrap.cc:492] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:10.197211 15481 ts_tablet_manager.cc:1367] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.007s user 0.001s sys 0.002s
I20240503 02:13:10.198571 15481 raft_consensus.cc:348] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.199115 15481 raft_consensus.cc:374] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.199266 15481 raft_consensus.cc:729] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.200536 15481 consensus_queue.cc:260] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.202052 15481 ts_tablet_manager.cc:1398] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:10.203737 15483 tablet_bootstrap.cc:492] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:10.203979 15483 ts_tablet_manager.cc:1367] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.006s
I20240503 02:13:10.205302 15483 raft_consensus.cc:348] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.205821 15483 raft_consensus.cc:374] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.206058 15483 raft_consensus.cc:729] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.206463 15483 consensus_queue.cc:260] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.207490 15483 ts_tablet_manager.cc:1398] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:10.210496 15482 tablet_bootstrap.cc:492] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:10.210852 15482 ts_tablet_manager.cc:1367] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.013s
I20240503 02:13:10.212275 15482 raft_consensus.cc:348] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.214169 15482 raft_consensus.cc:374] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.214337 15482 raft_consensus.cc:729] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.214812 15482 consensus_queue.cc:260] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.215771 15482 ts_tablet_manager.cc:1398] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:10.252884 15496 raft_consensus.cc:1050] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:10.253865 15496 consensus_queue.cc:237] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:10.260299 13302 raft_consensus.cc:1261] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:10.261109 15248 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:10.264493 10224 raft_consensus.cc:1261] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:10.264726 10357 raft_consensus.cc:1261] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:10.265396 15446 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:10.266012 15446 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:10.300252 15556 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:10.301012 10226 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:10.312153 10357 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:10.319082 15298 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.321208 9943 catalog_manager.cc:5484] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:10.335636 13305 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:10.353417 10491 consensus_queue.cc:237] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:10.356954 13303 raft_consensus.cc:1261] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:10.358943 10224 raft_consensus.cc:1261] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:10.368371 15446 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:10.368850 15496 consensus_queue.cc:1035] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:10.373328 15446 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:10.378684 10224 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:10.380189 15437 consensus_queue.cc:1035] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.394737 15311 raft_consensus.cc:482] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.395030 15311 raft_consensus.cc:504] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.396020 15311 leader_election.cc:288] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:10.403194 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4f280f95e467f98f907b02adfaaf0" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:10.403525 13168 raft_consensus.cc:2454] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:10.405750 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4f280f95e467f98f907b02adfaaf0" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:10.406069 13039 raft_consensus.cc:2454] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:10.406709 12843 leader_election.cc:302] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:10.407665 15533 raft_consensus.cc:2781] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:10.407790 15533 raft_consensus.cc:482] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:10.407920 15533 raft_consensus.cc:3037] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.411340 15533 raft_consensus.cc:504] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.415788 15437 consensus_queue.cc:1035] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.418325 15533 leader_election.cc:288] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:10.419409 13302 raft_consensus.cc:2932] T 31f72bc850c74289aa3a59ecc3dcc287 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:10.425149 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4f280f95e467f98f907b02adfaaf0" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:10.425457 13168 raft_consensus.cc:3037] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.426137 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 31f72bc850c74289aa3a59ecc3dcc287 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:10.428840 13168 raft_consensus.cc:2454] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:10.432058 15301 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.434253 12843 leader_election.cc:302] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:10.436167 15533 raft_consensus.cc:2781] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:10.436393 15533 raft_consensus.cc:686] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:10.436837 15533 consensus_queue.cc:237] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:10.439539 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bde4f280f95e467f98f907b02adfaaf0" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:10.439862 13040 raft_consensus.cc:3037] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:10.440045 9950 catalog_manager.cc:5484] T 31f72bc850c74289aa3a59ecc3dcc287 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:10.443969 13040 raft_consensus.cc:2454] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:10.454195 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 31f72bc850c74289aa3a59ecc3dcc287 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:10.446120 15474 raft_consensus.cc:979] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: : Instructing follower 8ee3fd5ad3624ea1b9f681d24e9aabf2 to start an election
I20240503 02:13:10.463141 15474 raft_consensus.cc:1067] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Signalling peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 to start an election
I20240503 02:13:10.469305 13302 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4"
dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
from {username='slave'} at 127.4.5.70:34829
I20240503 02:13:10.471422 13302 raft_consensus.cc:482] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:10.471879 13302 raft_consensus.cc:3037] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:10.478788 15582 tablet_replica.cc:330] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:10.479486 15582 raft_consensus.cc:2229] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:10.481585 15582 raft_consensus.cc:2258] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:10.479892 13302 raft_consensus.cc:504] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:10.484127 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:10.484405 13168 raft_consensus.cc:3037] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:10.488095 13168 raft_consensus.cc:2454] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 2.
I20240503 02:13:10.491677 9943 catalog_manager.cc:5484] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:10.492343 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:10.492673 13040 raft_consensus.cc:3032] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:10.492774 13040 raft_consensus.cc:729] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:10.493199 13040 consensus_queue.cc:260] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:10.493713 13040 raft_consensus.cc:3037] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:10.497272 13040 raft_consensus.cc:2454] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 2.
I20240503 02:13:10.497821 13246 leader_election.cc:302] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:10.498281 15437 raft_consensus.cc:2781] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:10.498466 15437 raft_consensus.cc:686] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:10.498812 15437 consensus_queue.cc:237] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:10.502935 15582 ts_tablet_manager.cc:1875] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:10.520807 10094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e3bba75394f24d4c91562089abad24c4" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:10.521157 10094 raft_consensus.cc:3037] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:10.524415 10094 raft_consensus.cc:2454] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 2.
I20240503 02:13:10.527985 13302 leader_election.cc:288] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:10.533843 9943 catalog_manager.cc:5484] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 1 to 2, leader changed from 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 2 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:10.536130 15059 consensus_queue.cc:1035] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.555094 15582 ts_tablet_manager.cc:1888] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:10.559358 15582 log.cc:1196] T 31f72bc850c74289aa3a59ecc3dcc287 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/31f72bc850c74289aa3a59ecc3dcc287
I20240503 02:13:10.567245 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 31f72bc850c74289aa3a59ecc3dcc287 (table rebalance_test_table_7 [id=69fc9a935e4b457a8e67b5ca88503adc]) successfully deleted
I20240503 02:13:10.579952 15059 consensus_queue.cc:1035] T 16f89ddc03524fa2bb6e5c66211564c0 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.621632 10224 raft_consensus.cc:1203] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:10.643867 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:10.653697 13040 raft_consensus.cc:1261] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:10.653697 13174 raft_consensus.cc:1261] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:10.654666 15575 consensus_queue.cc:1035] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.655303 15575 consensus_queue.cc:1035] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:10.682603 9950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_32"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:10.696287 13020 tablet_service.cc:1467] Processing CreateTablet for tablet d2d4b31cd7334ed6b18288c1b6d3f958 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.696357 12886 tablet_service.cc:1467] Processing CreateTablet for tablet d2d4b31cd7334ed6b18288c1b6d3f958 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.696906 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d4b31cd7334ed6b18288c1b6d3f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.696909 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d4b31cd7334ed6b18288c1b6d3f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.697407 10337 tablet_service.cc:1467] Processing CreateTablet for tablet d2d4b31cd7334ed6b18288c1b6d3f958 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.697966 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d4b31cd7334ed6b18288c1b6d3f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.701007 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 3303da732fa142b482c7d530c413c5dc (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.701594 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3303da732fa142b482c7d530c413c5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.702246 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 727b42f4ac0f45d9bf7c445f99fc3630 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.702567 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 3303da732fa142b482c7d530c413c5dc (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.702823 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727b42f4ac0f45d9bf7c445f99fc3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.703102 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3303da732fa142b482c7d530c413c5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.708597 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 727b42f4ac0f45d9bf7c445f99fc3630 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.709208 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727b42f4ac0f45d9bf7c445f99fc3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.725677 9946 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet bde4f280f95e467f98f907b02adfaaf0 (table rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e]). Expected version 1 got 0
I20240503 02:13:10.724058 15606 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:10.731196 15606 tablet_bootstrap.cc:654] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.738600 15608 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:10.739238 15608 tablet.cc:1620] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:10.747296 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 727b42f4ac0f45d9bf7c445f99fc3630 (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.747905 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727b42f4ac0f45d9bf7c445f99fc3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.748849 9943 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet bde4f280f95e467f98f907b02adfaaf0 (table rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e]). Expected version 1 got 0
I20240503 02:13:10.752952 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 3303da732fa142b482c7d530c413c5dc (DEFAULT_TABLE table=rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:10.753527 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3303da732fa142b482c7d530c413c5dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:10.763835 15610 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:10.764555 15610 tablet.cc:1620] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:10.771713 15611 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:10.778999 15611 tablet_bootstrap.cc:654] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.788129 15605 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:10.790215 15605 tablet_bootstrap.cc:654] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.791440 15611 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:10.791712 15611 ts_tablet_manager.cc:1367] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20240503 02:13:10.792719 15611 raft_consensus.cc:348] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:10.794519 15611 raft_consensus.cc:374] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.795861 15611 raft_consensus.cc:729] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.796262 15611 consensus_queue.cc:260] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:10.797012 15611 ts_tablet_manager.cc:1398] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:10.824362 15616 tablet_bootstrap.cc:492] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:10.830914 15616 tablet_bootstrap.cc:654] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.847008 9943 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e] alter complete (version 1)
I20240503 02:13:10.878962 15607 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:10.881522 15607 tablet_bootstrap.cc:654] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.888513 15605 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:10.888834 15605 ts_tablet_manager.cc:1367] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.101s user 0.012s sys 0.021s
I20240503 02:13:10.890259 15605 raft_consensus.cc:348] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:10.890640 15605 raft_consensus.cc:374] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.890812 15605 raft_consensus.cc:729] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.891153 15605 consensus_queue.cc:260] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:10.891724 15619 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:10.893172 15605 ts_tablet_manager.cc:1398] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:10.893483 15605 tablet_bootstrap.cc:492] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:10.874539 15618 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:10.897476 15618 tablet.cc:1620] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:10.898021 14793 raft_consensus.cc:482] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:10.898208 14793 raft_consensus.cc:504] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:10.900987 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727b42f4ac0f45d9bf7c445f99fc3630" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
W20240503 02:13:10.901815 10026 leader_election.cc:341] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:10.902042 15606 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:10.902220 15606 ts_tablet_manager.cc:1367] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.178s user 0.012s sys 0.013s
I20240503 02:13:10.902228 14793 leader_election.cc:288] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:10.903982 15605 tablet_bootstrap.cc:654] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.905486 15606 raft_consensus.cc:348] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:10.905786 15606 raft_consensus.cc:374] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.905900 15606 raft_consensus.cc:729] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.906601 15606 consensus_queue.cc:260] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:10.909754 15606 ts_tablet_manager.cc:1398] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.007s user 0.002s sys 0.002s
I20240503 02:13:10.914997 9950 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet bde4f280f95e467f98f907b02adfaaf0 (table rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e]). Expected version 1 got 0
I20240503 02:13:10.913522 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727b42f4ac0f45d9bf7c445f99fc3630" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
W20240503 02:13:10.916395 10029 leader_election.cc:341] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:10.916702 10029 leader_election.cc:302] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7; no voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:10.917631 14793 raft_consensus.cc:2726] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:10.922758 15607 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:10.923058 15607 ts_tablet_manager.cc:1367] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.003s
I20240503 02:13:10.925621 15607 raft_consensus.cc:348] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:10.926740 15605 tablet_bootstrap.cc:492] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:10.927328 15605 ts_tablet_manager.cc:1367] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.001s
I20240503 02:13:10.928535 15605 raft_consensus.cc:348] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:10.928889 15605 raft_consensus.cc:374] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.929033 15605 raft_consensus.cc:729] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.932276 15619 tablet_bootstrap.cc:654] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.936612 15619 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:10.937310 15607 raft_consensus.cc:374] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.937372 15619 ts_tablet_manager.cc:1367] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.046s user 0.005s sys 0.000s
I20240503 02:13:10.937568 15605 consensus_queue.cc:260] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:10.938282 15619 raft_consensus.cc:348] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:10.937988 15607 raft_consensus.cc:729] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.939405 15605 ts_tablet_manager.cc:1398] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:13:10.939669 15607 consensus_queue.cc:260] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:10.941074 15607 ts_tablet_manager.cc:1398] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:13:10.945607 15607 tablet_bootstrap.cc:492] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:10.947845 15607 tablet_bootstrap.cc:654] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:10.939846 15619 raft_consensus.cc:374] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:10.953493 15619 raft_consensus.cc:729] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:10.953815 15619 consensus_queue.cc:260] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:10.954512 15619 ts_tablet_manager.cc:1398] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.017s user 0.001s sys 0.000s
I20240503 02:13:10.956749 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:10.964936 15616 tablet_bootstrap.cc:492] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:10.970809 15616 ts_tablet_manager.cc:1367] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.147s user 0.006s sys 0.005s
I20240503 02:13:10.982228 15608 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:10.982929 15608 tablet.cc:1620] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:10.995230 15610 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:10.996023 15610 tablet.cc:1620] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:11.010553 10094 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:11.012077 15438 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:11.029263 15616 raft_consensus.cc:348] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:11.029646 15616 raft_consensus.cc:374] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.029769 15616 raft_consensus.cc:729] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.030079 15616 consensus_queue.cc:260] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:11.031507 15616 ts_tablet_manager.cc:1398] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20240503 02:13:11.043619 15496 raft_consensus.cc:482] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.043874 15496 raft_consensus.cc:504] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:11.044764 15496 leader_election.cc:288] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:11.045485 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727b42f4ac0f45d9bf7c445f99fc3630" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:11.045840 10091 raft_consensus.cc:2454] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:13:11.048079 10426 leader_election.cc:302] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:11.049206 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727b42f4ac0f45d9bf7c445f99fc3630" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:11.053929 13172 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:11.055310 15437 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:11.056887 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e] alter complete (version 2)
W20240503 02:13:11.060724 10427 leader_election.cc:341] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:11.065236 15248 raft_consensus.cc:2781] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.065393 15248 raft_consensus.cc:482] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.065501 15248 raft_consensus.cc:3037] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.070860 15438 raft_consensus.cc:1050] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:11.072083 15438 consensus_queue.cc:237] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:11.072744 15248 raft_consensus.cc:504] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:11.075377 15248 leader_election.cc:288] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:11.076588 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727b42f4ac0f45d9bf7c445f99fc3630" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:11.077288 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727b42f4ac0f45d9bf7c445f99fc3630" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:11.077595 10091 raft_consensus.cc:3037] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:13:11.080718 10427 leader_election.cc:341] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:11.085237 10091 raft_consensus.cc:2454] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:11.088611 10426 leader_election.cc:302] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters: f93b103af083471b8e8c0376e9148aad
I20240503 02:13:11.088882 15618 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:11.089290 15248 raft_consensus.cc:2781] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.089488 15618 tablet.cc:1620] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:11.089507 15248 raft_consensus.cc:686] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:11.089856 15248 consensus_queue.cc:237] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:11.032341 15616 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:11.092854 9950 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet bde4f280f95e467f98f907b02adfaaf0 (table rebalancer_dynamic_table [id=3ef8864e570f4561b7b974455aa0f57e]). Expected version 2 got 1
I20240503 02:13:11.103382 15616 tablet_bootstrap.cc:654] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.112430 13040 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:11.113644 15437 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:11.145151 15607 tablet_bootstrap.cc:492] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:11.145411 15607 ts_tablet_manager.cc:1367] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.200s user 0.019s sys 0.052s
I20240503 02:13:11.145956 15616 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:11.146215 15616 ts_tablet_manager.cc:1367] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.114s user 0.001s sys 0.003s
I20240503 02:13:11.148662 15616 raft_consensus.cc:348] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:11.149474 15616 raft_consensus.cc:374] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.149613 15616 raft_consensus.cc:729] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.149930 15616 consensus_queue.cc:260] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:11.150060 15607 raft_consensus.cc:348] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:11.150337 15607 raft_consensus.cc:374] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.152256 15607 raft_consensus.cc:729] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.153062 15607 consensus_queue.cc:260] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:11.155799 15616 ts_tablet_manager.cc:1398] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20240503 02:13:11.159842 10226 raft_consensus.cc:1261] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:11.160513 13174 raft_consensus.cc:1261] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:11.160779 15437 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:11.161415 15438 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:11.164384 13302 consensus_queue.cc:237] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:11.167017 10091 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:11.167680 15437 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:11.168396 13172 raft_consensus.cc:1261] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:11.169198 15515 consensus_queue.cc:1035] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:11.170172 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:11.171072 13040 raft_consensus.cc:1261] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:11.174026 15437 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:11.177582 13172 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:11.181561 15515 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:11.184891 10094 raft_consensus.cc:2932] T e3bba75394f24d4c91562089abad24c4 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:11.187486 9950 catalog_manager.cc:5484] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:11.191979 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e3bba75394f24d4c91562089abad24c4 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:11.196429 9952 catalog_manager.cc:5484] T e3bba75394f24d4c91562089abad24c4 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 5, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 2 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:11.211623 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet e3bba75394f24d4c91562089abad24c4 with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:11.222225 15311 raft_consensus.cc:482] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.222441 15311 raft_consensus.cc:504] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:11.223378 15311 leader_election.cc:288] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:11.224130 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3303da732fa142b482c7d530c413c5dc" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:13:11.224472 10354 raft_consensus.cc:2454] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:11.224586 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3303da732fa142b482c7d530c413c5dc" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:11.224884 13168 raft_consensus.cc:2454] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:11.225049 12842 leader_election.cc:302] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:11.225394 15311 raft_consensus.cc:2781] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.225535 15311 raft_consensus.cc:482] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.225641 15311 raft_consensus.cc:3037] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.228142 15063 raft_consensus.cc:482] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.228327 15063 raft_consensus.cc:504] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:11.228638 15607 ts_tablet_manager.cc:1398] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.083s user 0.003s sys 0.000s
I20240503 02:13:11.229489 15063 leader_election.cc:288] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:11.230265 15311 raft_consensus.cc:504] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:11.230821 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4b31cd7334ed6b18288c1b6d3f958" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:11.231202 12904 raft_consensus.cc:2454] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:11.231462 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3303da732fa142b482c7d530c413c5dc" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:11.231679 10354 raft_consensus.cc:3037] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.232300 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4b31cd7334ed6b18288c1b6d3f958" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:11.232666 13039 raft_consensus.cc:2454] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:11.233211 10292 leader_election.cc:302] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:11.233222 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3303da732fa142b482c7d530c413c5dc" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:11.233552 13174 raft_consensus.cc:3037] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.233682 15346 raft_consensus.cc:2781] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.233803 15346 raft_consensus.cc:482] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.233944 15346 raft_consensus.cc:3037] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.234885 10354 raft_consensus.cc:2454] T 3303da732fa142b482c7d530c413c5dc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:11.235854 12842 leader_election.cc:302] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:11.236202 15575 raft_consensus.cc:2781] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.236344 15575 raft_consensus.cc:686] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:11.236718 15575 consensus_queue.cc:237] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:11.238677 13174 raft_consensus.cc:2454] T 3303da732fa142b482c7d530c413c5dc P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:11.239746 15311 leader_election.cc:288] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:11.240676 15645 tablet_replica.cc:330] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:11.241102 15645 raft_consensus.cc:2229] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.241422 15645 raft_consensus.cc:2258] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.247608 15346 raft_consensus.cc:504] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:11.249140 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4b31cd7334ed6b18288c1b6d3f958" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:11.249351 13039 raft_consensus.cc:3037] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.255918 15645 ts_tablet_manager.cc:1875] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:11.262507 13039 raft_consensus.cc:2454] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:11.265780 10295 leader_election.cc:302] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, 8d454a3607d042dc9559ed59c3ccd84b; no voters:
I20240503 02:13:11.266454 15613 raft_consensus.cc:2781] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.266615 15613 raft_consensus.cc:686] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:13:11.267021 15613 consensus_queue.cc:237] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:11.276623 9950 catalog_manager.cc:5484] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:11.278230 15346 leader_election.cc:288] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:11.279728 9947 catalog_manager.cc:5484] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:11.285830 12907 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4b31cd7334ed6b18288c1b6d3f958" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:11.286132 12907 raft_consensus.cc:3037] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.286785 15645 ts_tablet_manager.cc:1888] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 02:13:11.286944 15645 log.cc:1196] T e3bba75394f24d4c91562089abad24c4 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/e3bba75394f24d4c91562089abad24c4
I20240503 02:13:11.289949 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet e3bba75394f24d4c91562089abad24c4 (table rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]) successfully deleted
I20240503 02:13:11.302603 12907 raft_consensus.cc:2454] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:11.324466 15608 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:11.326404 15610 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:11.342293 9947 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=3ef8864e570f4561b7b974455aa0f57e] alter complete (version 3)
I20240503 02:13:11.343195 15618 tablet.cc:1722] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
W20240503 02:13:11.347491 13246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 155024us
I20240503 02:13:11.352355 15437 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:11.358342 10226 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:11.365837 9947 catalog_manager.cc:5484] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:11.367107 13168 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:11.377403 13302 consensus_queue.cc:237] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:11.377671 13040 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:11.378161 15590 raft_consensus.cc:482] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.378351 15590 raft_consensus.cc:504] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:11.379344 15590 leader_election.cc:288] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:11.379892 10226 raft_consensus.cc:1261] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:11.379953 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727b42f4ac0f45d9bf7c445f99fc3630" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:11.380295 10091 raft_consensus.cc:2379] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f93b103af083471b8e8c0376e9148aad in current term 1: Already voted for candidate c6ec2f1283994a3089741715f811c850 in this term.
I20240503 02:13:11.381095 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "727b42f4ac0f45d9bf7c445f99fc3630" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:11.381335 15643 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:11.382057 13107 leader_election.cc:302] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850
I20240503 02:13:11.382931 15590 raft_consensus.cc:3037] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.387179 15515 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:11.389693 10226 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:11.400905 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 83f291bfdc064894ae0f7d6fc58655df with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:11.405508 9947 catalog_manager.cc:5484] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:11.415810 15590 raft_consensus.cc:2726] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:11.416090 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f291bfdc064894ae0f7d6fc58655df with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:11.416671 13168 raft_consensus.cc:1261] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:11.418588 15643 consensus_queue.cc:1035] T 83f291bfdc064894ae0f7d6fc58655df P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:11.420364 13172 raft_consensus.cc:2932] T 83f291bfdc064894ae0f7d6fc58655df P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:11.424784 15645 tablet_replica.cc:330] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:11.425082 15645 raft_consensus.cc:2229] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.425370 15645 raft_consensus.cc:2258] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.426937 15645 ts_tablet_manager.cc:1875] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:11.439529 15645 ts_tablet_manager.cc:1888] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:11.439805 15645 log.cc:1196] T 83f291bfdc064894ae0f7d6fc58655df P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/83f291bfdc064894ae0f7d6fc58655df
I20240503 02:13:11.440979 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 83f291bfdc064894ae0f7d6fc58655df (table rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]) successfully deleted
W20240503 02:13:11.448078 12973 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.69:33471, user_credentials={real_user=slave}} blocked reactor thread for 63993.1us
I20240503 02:13:11.453049 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 83f291bfdc064894ae0f7d6fc58655df with delete_type TABLET_DATA_TOMBSTONED (Replica with old config index 3 (current committed config index is 4)) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:11.453800 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 83f291bfdc064894ae0f7d6fc58655df (table rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]) successfully deleted
I20240503 02:13:11.466993 9950 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:11.467357 9950 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:11.479589 15496 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.481307 9950 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet bde4f280f95e467f98f907b02adfaaf0
I20240503 02:13:11.482445 12886 tablet_service.cc:1514] Processing DeleteTablet for tablet bde4f280f95e467f98f907b02adfaaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:11 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:11.482589 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet bde4f280f95e467f98f907b02adfaaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:11 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:11.485795 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet bde4f280f95e467f98f907b02adfaaf0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:11 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:11.486250 15645 tablet_replica.cc:330] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:11.486568 15645 raft_consensus.cc:2229] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.486897 15645 raft_consensus.cc:2258] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.488565 15645 ts_tablet_manager.cc:1875] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.524259 15645 ts_tablet_manager.cc:1888] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:11.528342 15446 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20240503 02:13:11.529418 15645 log.cc:1196] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/bde4f280f95e467f98f907b02adfaaf0
I20240503 02:13:11.530762 15645 ts_tablet_manager.cc:1909] T bde4f280f95e467f98f907b02adfaaf0 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:11.531934 15660 tablet_replica.cc:330] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:11.531968 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet bde4f280f95e467f98f907b02adfaaf0 (table rebalancer_dynamic_table_new_name [id=3ef8864e570f4561b7b974455aa0f57e]) successfully deleted
I20240503 02:13:11.532203 15660 raft_consensus.cc:2229] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:11.532717 15660 raft_consensus.cc:2258] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.533802 15660 ts_tablet_manager.cc:1875] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.535224 15661 tablet_replica.cc:330] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:11.537834 15661 raft_consensus.cc:2229] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:11.538370 15661 raft_consensus.cc:2258] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:11.543694 15661 ts_tablet_manager.cc:1875] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:11.566290 15660 ts_tablet_manager.cc:1888] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:11.580667 15660 log.cc:1196] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/bde4f280f95e467f98f907b02adfaaf0
I20240503 02:13:11.583787 15661 ts_tablet_manager.cc:1888] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:11.583993 15661 log.cc:1196] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/bde4f280f95e467f98f907b02adfaaf0
I20240503 02:13:11.584647 15661 ts_tablet_manager.cc:1909] T bde4f280f95e467f98f907b02adfaaf0 P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:13:11.585808 9950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:11.581501 15660 ts_tablet_manager.cc:1909] T bde4f280f95e467f98f907b02adfaaf0 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:13:11.597249 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet bde4f280f95e467f98f907b02adfaaf0 (table rebalancer_dynamic_table_new_name [id=3ef8864e570f4561b7b974455aa0f57e]) successfully deleted
I20240503 02:13:11.600883 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 909955a870f54174ab44cfcf5f9b5bc0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f]), partition=
I20240503 02:13:11.601356 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909955a870f54174ab44cfcf5f9b5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.601375 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 909955a870f54174ab44cfcf5f9b5bc0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f]), partition=
I20240503 02:13:11.601837 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909955a870f54174ab44cfcf5f9b5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.605441 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet bde4f280f95e467f98f907b02adfaaf0 (table rebalancer_dynamic_table_new_name [id=3ef8864e570f4561b7b974455aa0f57e]) successfully deleted
I20240503 02:13:11.605847 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 909955a870f54174ab44cfcf5f9b5bc0 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f]), partition=
I20240503 02:13:11.606279 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 909955a870f54174ab44cfcf5f9b5bc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.618772 15616 tablet_bootstrap.cc:492] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:11.621140 15616 tablet_bootstrap.cc:654] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.660207 15669 tablet_bootstrap.cc:492] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:11.660581 15668 tablet_bootstrap.cc:492] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:11.662233 15668 tablet_bootstrap.cc:654] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.663918 15669 tablet_bootstrap.cc:654] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.681274 15669 tablet_bootstrap.cc:492] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:11.682029 15669 ts_tablet_manager.cc:1367] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.003s
I20240503 02:13:11.683290 15669 raft_consensus.cc:348] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.683768 15669 raft_consensus.cc:374] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.683941 15669 raft_consensus.cc:729] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.685554 15668 tablet_bootstrap.cc:492] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:11.685802 15668 ts_tablet_manager.cc:1367] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.000s
I20240503 02:13:11.686990 15668 raft_consensus.cc:348] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.687357 15668 raft_consensus.cc:374] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.687536 15668 raft_consensus.cc:729] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.687961 15668 consensus_queue.cc:260] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.689220 15616 tablet_bootstrap.cc:492] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:11.689420 15616 ts_tablet_manager.cc:1367] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.009s
I20240503 02:13:11.690506 15616 raft_consensus.cc:348] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.690757 15616 raft_consensus.cc:374] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:11.690855 15616 raft_consensus.cc:729] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:11.691138 15616 consensus_queue.cc:260] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.691713 15616 ts_tablet_manager.cc:1398] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20240503 02:13:11.702325 15668 ts_tablet_manager.cc:1398] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20240503 02:13:11.684295 15669 consensus_queue.cc:260] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.703622 15669 ts_tablet_manager.cc:1398] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.021s user 0.002s sys 0.000s
I20240503 02:13:11.728472 15248 raft_consensus.cc:482] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:11.728780 15248 raft_consensus.cc:504] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.729743 15248 leader_election.cc:288] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:11.732324 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909955a870f54174ab44cfcf5f9b5bc0" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:11.732676 13302 raft_consensus.cc:2454] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:13:11.733192 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909955a870f54174ab44cfcf5f9b5bc0" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:11.733500 13168 raft_consensus.cc:2454] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:13:11.734076 10427 leader_election.cc:302] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:11.734687 15446 raft_consensus.cc:2781] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:11.734795 15446 raft_consensus.cc:482] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:11.734894 15446 raft_consensus.cc:3037] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.738143 15446 raft_consensus.cc:504] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.739974 15446 leader_election.cc:288] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:11.740973 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909955a870f54174ab44cfcf5f9b5bc0" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:11.741304 13302 raft_consensus.cc:3037] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.745663 13302 raft_consensus.cc:2454] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:11.745764 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "909955a870f54174ab44cfcf5f9b5bc0" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:11.746117 13168 raft_consensus.cc:3037] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:11.746338 10424 leader_election.cc:302] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:11.746960 15446 raft_consensus.cc:2781] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:11.747578 15446 raft_consensus.cc:686] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:11.747915 15446 consensus_queue.cc:237] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:11.760371 9943 catalog_manager.cc:5484] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:11.767758 13168 raft_consensus.cc:2454] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:11.811034 15346 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.815282 10091 consensus_queue.cc:237] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:11.822985 13302 raft_consensus.cc:1261] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:11.823746 13168 raft_consensus.cc:1261] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:11.824525 14793 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:11.825224 14793 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:11.826313 10026 consensus_peers.cc:485] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 432c8c5d42cb454fb7d1ca5af4782bb7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:11.835716 15613 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.842919 15671 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:11.845418 13168 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:11.847647 15575 consensus_queue.cc:1035] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.857993 15575 consensus_queue.cc:1035] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.861639 13302 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:11.872676 9950 catalog_manager.cc:5484] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:11.886565 9947 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:11.899019 9950 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:11.913830 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 09468c67719244c18d200463892a4f84 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.914407 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09468c67719244c18d200463892a4f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.918433 13289 tablet_service.cc:1467] Processing CreateTablet for tablet 189d7bee0d6f40a69f25aa578f3d1dec (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.918859 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 189d7bee0d6f40a69f25aa578f3d1dec (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.919093 13289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189d7bee0d6f40a69f25aa578f3d1dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.919452 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189d7bee0d6f40a69f25aa578f3d1dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.923509 13288 tablet_service.cc:1467] Processing CreateTablet for tablet fde14f2778bc4b7f830bb7bdcbb6afd1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.924093 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde14f2778bc4b7f830bb7bdcbb6afd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.925716 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 09468c67719244c18d200463892a4f84 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.928685 15669 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:11.929126 13019 tablet_service.cc:1467] Processing CreateTablet for tablet fde14f2778bc4b7f830bb7bdcbb6afd1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.929672 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde14f2778bc4b7f830bb7bdcbb6afd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.930753 15669 tablet_bootstrap.cc:654] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.933220 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09468c67719244c18d200463892a4f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.938926 13153 tablet_service.cc:1467] Processing CreateTablet for tablet fde14f2778bc4b7f830bb7bdcbb6afd1 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.939641 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fde14f2778bc4b7f830bb7bdcbb6afd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.949373 13302 raft_consensus.cc:1261] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:11.950400 15248 consensus_queue.cc:1035] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.954154 13168 raft_consensus.cc:1261] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:11.955288 15446 consensus_queue.cc:1035] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:11.982021 15717 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:11.984344 15717 tablet_bootstrap.cc:654] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.988879 15720 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:11.989725 15720 tablet.cc:1620] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:13:11.990644 15721 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:11.991221 15722 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:11.992035 15616 tablet_bootstrap.cc:492] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:11.993770 9947 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 909955a870f54174ab44cfcf5f9b5bc0 (table rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f]). Expected version 1 got 0
I20240503 02:13:11.994203 15616 tablet_bootstrap.cc:654] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:11.997498 12886 tablet_service.cc:1467] Processing CreateTablet for tablet 09468c67719244c18d200463892a4f84 (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.998577 12886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09468c67719244c18d200463892a4f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:11.998827 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 189d7bee0d6f40a69f25aa578f3d1dec (DEFAULT_TABLE table=rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:11.999425 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189d7bee0d6f40a69f25aa578f3d1dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.003186 9943 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 909955a870f54174ab44cfcf5f9b5bc0 (table rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f]). Expected version 1 got 0
W20240503 02:13:12.008598 12843 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 99227.8us
I20240503 02:13:12.014712 15725 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:12.016050 15725 tablet.cc:1620] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:13:12.016399 9950 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet 909955a870f54174ab44cfcf5f9b5bc0 (table rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f]). Expected version 1 got 0
I20240503 02:13:12.020538 9947 catalog_manager.cc:5576] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671) does not have the latest schema for tablet 909955a870f54174ab44cfcf5f9b5bc0 (table rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f]). Expected version 1 got 0
I20240503 02:13:12.024500 15722 tablet.cc:1620] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:12.028357 15721 tablet_bootstrap.cc:654] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.026384 15616 tablet_bootstrap.cc:492] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:12.029253 15616 ts_tablet_manager.cc:1367] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.007s
I20240503 02:13:12.029644 15717 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:12.029920 15717 ts_tablet_manager.cc:1367] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.048s user 0.005s sys 0.000s
I20240503 02:13:12.030506 15616 raft_consensus.cc:348] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.031044 15717 raft_consensus.cc:348] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.031201 15616 raft_consensus.cc:374] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.031318 15616 raft_consensus.cc:729] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.031455 15717 raft_consensus.cc:374] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.031615 15717 raft_consensus.cc:729] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.031700 15616 consensus_queue.cc:260] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.019425 15669 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:12.031924 15717 consensus_queue.cc:260] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.032328 15669 ts_tablet_manager.cc:1367] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.104s user 0.007s sys 0.006s
I20240503 02:13:12.032703 15717 ts_tablet_manager.cc:1398] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:12.033053 15717 tablet_bootstrap.cc:492] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:12.033924 15669 raft_consensus.cc:348] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.034253 15669 raft_consensus.cc:374] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.034335 15669 raft_consensus.cc:729] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.034700 15669 consensus_queue.cc:260] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.035059 15717 tablet_bootstrap.cc:654] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.035521 15669 ts_tablet_manager.cc:1398] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:12.043545 15717 tablet_bootstrap.cc:492] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:12.043916 15717 ts_tablet_manager.cc:1367] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20240503 02:13:12.045082 15726 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:12.045308 15669 tablet_bootstrap.cc:492] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:12.048482 15726 tablet_bootstrap.cc:654] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.049796 15717 raft_consensus.cc:348] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.051206 15717 raft_consensus.cc:374] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.051584 15717 raft_consensus.cc:729] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.052218 15717 consensus_queue.cc:260] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.052356 15669 tablet_bootstrap.cc:654] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.057245 15717 ts_tablet_manager.cc:1398] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20240503 02:13:12.064759 15726 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:12.065353 15726 ts_tablet_manager.cc:1367] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.008s
I20240503 02:13:12.069408 15726 raft_consensus.cc:348] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.069875 15726 raft_consensus.cc:374] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.070289 15726 raft_consensus.cc:729] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.070861 15726 consensus_queue.cc:260] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.072136 15726 ts_tablet_manager.cc:1398] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20240503 02:13:12.080927 9950 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f] alter complete (version 1)
I20240503 02:13:12.081573 15726 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:12.084043 15726 tablet_bootstrap.cc:654] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.096283 15616 ts_tablet_manager.cc:1398] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.067s user 0.001s sys 0.001s
I20240503 02:13:12.092491 10224 consensus_queue.cc:237] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:12.112663 15669 tablet_bootstrap.cc:492] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:12.113191 15669 ts_tablet_manager.cc:1367] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.002s
I20240503 02:13:12.114311 15669 raft_consensus.cc:348] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.114684 15669 raft_consensus.cc:374] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.114836 15669 raft_consensus.cc:729] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.115190 15669 consensus_queue.cc:260] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.116083 15669 ts_tablet_manager.cc:1398] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20240503 02:13:12.116570 15669 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:12.119606 15669 tablet_bootstrap.cc:654] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.135190 15721 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:12.135396 15721 ts_tablet_manager.cc:1367] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.145s user 0.004s sys 0.000s
I20240503 02:13:12.136122 15721 raft_consensus.cc:348] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.136530 15721 raft_consensus.cc:374] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.136658 15721 raft_consensus.cc:729] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.138293 15721 consensus_queue.cc:260] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.139178 15721 ts_tablet_manager.cc:1398] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:12.144733 15726 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:12.144872 10354 raft_consensus.cc:1261] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:12.145002 15726 ts_tablet_manager.cc:1367] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.072s user 0.005s sys 0.012s
I20240503 02:13:12.145700 15629 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:12.146170 15726 raft_consensus.cc:348] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.146432 15726 raft_consensus.cc:374] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.146525 15726 raft_consensus.cc:729] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.146975 15726 consensus_queue.cc:260] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.148850 15726 ts_tablet_manager.cc:1398] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20240503 02:13:12.149530 15669 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:12.149791 15669 ts_tablet_manager.cc:1367] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.003s
W20240503 02:13:12.150542 10159 consensus_peers.cc:485] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 674ea54f2cbd4c438cab5a7723c5a2c6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:12.151157 15669 raft_consensus.cc:348] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.151538 15669 raft_consensus.cc:374] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.151634 15669 raft_consensus.cc:729] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.151964 10489 raft_consensus.cc:1261] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:12.152022 15669 consensus_queue.cc:260] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.152729 15629 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:12.156986 15669 ts_tablet_manager.cc:1398] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20240503 02:13:12.160746 15515 raft_consensus.cc:482] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.161026 15515 raft_consensus.cc:504] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.161975 15515 leader_election.cc:288] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:12.163008 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09468c67719244c18d200463892a4f84" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:12.163354 12908 raft_consensus.cc:2454] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:12.163285 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09468c67719244c18d200463892a4f84" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:12.163581 13040 raft_consensus.cc:2454] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:12.164203 13246 leader_election.cc:302] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:12.164695 15301 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:12.166049 10354 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:12.169095 15515 raft_consensus.cc:2781] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.169282 15515 raft_consensus.cc:482] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.169409 15515 raft_consensus.cc:3037] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.171990 15515 raft_consensus.cc:504] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.173249 15515 leader_election.cc:288] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:12.173933 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09468c67719244c18d200463892a4f84" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:12.174192 13038 raft_consensus.cc:3037] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.174965 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09468c67719244c18d200463892a4f84" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:12.175271 12908 raft_consensus.cc:3037] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.175904 9943 catalog_manager.cc:5484] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:12.178284 12908 raft_consensus.cc:2454] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:12.178278 13038 raft_consensus.cc:2454] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:12.180394 13243 leader_election.cc:302] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:12.180902 15515 raft_consensus.cc:2781] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.182816 15515 raft_consensus.cc:686] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:12.183166 15515 consensus_queue.cc:237] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.188474 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:12.196031 10489 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:12.202035 15715 raft_consensus.cc:482] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.202301 15715 raft_consensus.cc:504] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.203604 15715 leader_election.cc:288] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:12.207337 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189d7bee0d6f40a69f25aa578f3d1dec" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:12.207630 10224 raft_consensus.cc:2454] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:12.208210 12842 leader_election.cc:302] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:12.208837 15715 raft_consensus.cc:2781] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.208976 15715 raft_consensus.cc:482] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.209066 15715 raft_consensus.cc:3037] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.209147 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189d7bee0d6f40a69f25aa578f3d1dec" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:12.209440 13305 raft_consensus.cc:2454] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:12.211968 15715 raft_consensus.cc:504] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.213660 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189d7bee0d6f40a69f25aa578f3d1dec" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:12.213781 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "189d7bee0d6f40a69f25aa578f3d1dec" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:12.213994 10224 raft_consensus.cc:3037] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.214105 13305 raft_consensus.cc:3037] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.216563 15715 leader_election.cc:288] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:12.223304 15437 raft_consensus.cc:482] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.223598 15437 raft_consensus.cc:504] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.224639 15437 leader_election.cc:288] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:12.225929 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde14f2778bc4b7f830bb7bdcbb6afd1" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:12.226279 13038 raft_consensus.cc:2454] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:12.226282 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde14f2778bc4b7f830bb7bdcbb6afd1" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:12.226574 13168 raft_consensus.cc:2454] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:12.227506 13246 leader_election.cc:302] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:12.228186 15437 raft_consensus.cc:2781] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.228317 15437 raft_consensus.cc:482] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.228401 15437 raft_consensus.cc:3037] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.229240 10224 raft_consensus.cc:2454] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:12.229984 12842 leader_election.cc:302] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:12.230643 15715 raft_consensus.cc:2781] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.232980 13305 raft_consensus.cc:2454] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:12.231835 15437 raft_consensus.cc:504] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.236959 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde14f2778bc4b7f830bb7bdcbb6afd1" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:12.237294 13168 raft_consensus.cc:3037] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.241504 15720 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:12.242244 15720 tablet.cc:1620] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:13:12.242938 15722 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:12.241365 13168 raft_consensus.cc:2454] T fde14f2778bc4b7f830bb7bdcbb6afd1 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:12.243546 15722 tablet.cc:1620] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:12.243840 13246 leader_election.cc:302] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:12.244165 15725 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:12.231375 15715 raft_consensus.cc:686] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:12.244670 15643 raft_consensus.cc:2781] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.244777 15725 tablet.cc:1620] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: MemRowSet was empty: no flush needed.
I20240503 02:13:12.244830 15643 raft_consensus.cc:686] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:12.244858 15715 consensus_queue.cc:237] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:12.245198 15643 consensus_queue.cc:237] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:12.249246 15437 leader_election.cc:288] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:12.235950 9947 catalog_manager.cc:5484] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:12.253649 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fde14f2778bc4b7f830bb7bdcbb6afd1" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:12.254061 13038 raft_consensus.cc:3037] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.259886 13038 raft_consensus.cc:2454] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:12.265556 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=9cbc24e726d34722b5ff535376da654f] alter complete (version 2)
I20240503 02:13:12.271306 9952 catalog_manager.cc:5484] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:12.295233 9946 catalog_manager.cc:5484] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:12.324776 10091 consensus_queue.cc:237] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:12.327621 10224 raft_consensus.cc:1261] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:12.327831 10354 raft_consensus.cc:1261] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:12.328583 15703 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:12.329111 14793 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:12.329475 10029 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2fadc7e9c7604b5d8f7b8751b6ca78a7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:12.333334 15699 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:12.334219 10354 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:12.342887 9943 catalog_manager.cc:5484] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:12.352728 10224 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:12.370970 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:12.373294 12908 consensus_queue.cc:237] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
W20240503 02:13:12.376708 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f1f1661a75a74746a4485b88b8e5b185. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:12.376937 10091 raft_consensus.cc:1261] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:12.377606 10489 raft_consensus.cc:1261] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:12.378451 15715 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:12.378857 15709 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:12.383141 15720 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:12.384631 15722 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:12.384897 15725 tablet.cc:1722] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:12.394887 15709 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:12.396507 10091 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:12.405156 9943 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=9cbc24e726d34722b5ff535376da654f] alter complete (version 3)
I20240503 02:13:12.415710 9943 catalog_manager.cc:5484] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:12.417213 10489 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:12.446164 15770 ts_tablet_manager.cc:902] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:12.456615 15770 tablet_copy_client.cc:308] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:12.457676 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 432c8c5d42cb454fb7d1ca5af4782bb7 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:56811)
I20240503 02:13:12.457913 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 432c8c5d42cb454fb7d1ca5af4782bb7 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:56811: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-432c8c5d42cb454fb7d1ca5af4782bb7
I20240503 02:13:12.463517 10115 tablet_copy_source_session.cc:205] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:12.463665 10227 consensus_queue.cc:237] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:12.465608 15770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 432c8c5d42cb454fb7d1ca5af4782bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.469135 10091 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:12.469175 10354 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:12.469769 10159 consensus_peers.cc:485] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13e83ce8820b491992c041a8c21fa3df. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:12.469972 15741 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:12.470335 15740 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:12.472620 15770 tablet_copy_client.cc:783] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:12.472921 15770 tablet_copy_client.cc:647] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:12.475028 15741 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:12.484733 15770 tablet_copy_client.cc:523] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:12.484165 9943 catalog_manager.cc:5484] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:12.489987 15770 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:12.495311 10091 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:12.511940 9943 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:12.512251 9943 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:12.513278 15770 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:12.513846 15770 tablet_bootstrap.cc:492] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:12.514146 15770 ts_tablet_manager.cc:1367] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.024s user 0.007s sys 0.004s
I20240503 02:13:12.516877 9943 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 909955a870f54174ab44cfcf5f9b5bc0
I20240503 02:13:12.517949 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 909955a870f54174ab44cfcf5f9b5bc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:12 UTC) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:12.518057 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 909955a870f54174ab44cfcf5f9b5bc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:12 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:12.518318 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet 909955a870f54174ab44cfcf5f9b5bc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:12 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:12.520673 15783 tablet_replica.cc:330] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:12.520925 15782 tablet_replica.cc:330] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:12.520979 15783 raft_consensus.cc:2229] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:12.521381 15782 raft_consensus.cc:2229] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:12.522059 15782 raft_consensus.cc:2258] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:12.522230 15783 raft_consensus.cc:2258] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:12.523998 15782 ts_tablet_manager.cc:1875] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:12.517632 15770 raft_consensus.cc:348] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:12.531147 15770 raft_consensus.cc:729] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:12.531514 15770 consensus_queue.cc:260] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:12.533159 15781 tablet_replica.cc:330] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:12.533622 15781 raft_consensus.cc:2229] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:12.535720 15770 ts_tablet_manager.cc:1398] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20240503 02:13:12.536195 15781 raft_consensus.cc:2258] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:12.537655 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-432c8c5d42cb454fb7d1ca5af4782bb7 received from {username='slave'} at 127.4.5.69:56811
I20240503 02:13:12.537824 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-432c8c5d42cb454fb7d1ca5af4782bb7 on tablet 432c8c5d42cb454fb7d1ca5af4782bb7 with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:12.537881 15783 ts_tablet_manager.cc:1875] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:12.541208 15781 ts_tablet_manager.cc:1875] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:12.551945 10357 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:12.557562 15781 ts_tablet_manager.cc:1888] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:12.560640 15781 log.cc:1196] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/909955a870f54174ab44cfcf5f9b5bc0
I20240503 02:13:12.561779 15781 ts_tablet_manager.cc:1909] T 909955a870f54174ab44cfcf5f9b5bc0 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting consensus metadata
I20240503 02:13:12.565655 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 909955a870f54174ab44cfcf5f9b5bc0 (table rebalancer_dynamic_table_new_name [id=9cbc24e726d34722b5ff535376da654f]) successfully deleted
I20240503 02:13:12.567255 15783 ts_tablet_manager.cc:1888] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:12.568465 15782 ts_tablet_manager.cc:1888] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:12.568745 15782 log.cc:1196] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/909955a870f54174ab44cfcf5f9b5bc0
I20240503 02:13:12.572939 15782 ts_tablet_manager.cc:1909] T 909955a870f54174ab44cfcf5f9b5bc0 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:13:12.574471 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 909955a870f54174ab44cfcf5f9b5bc0 (table rebalancer_dynamic_table_new_name [id=9cbc24e726d34722b5ff535376da654f]) successfully deleted
I20240503 02:13:12.574874 15783 log.cc:1196] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/909955a870f54174ab44cfcf5f9b5bc0
I20240503 02:13:12.575414 15783 ts_tablet_manager.cc:1909] T 909955a870f54174ab44cfcf5f9b5bc0 P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:13:12.576696 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 909955a870f54174ab44cfcf5f9b5bc0 (table rebalancer_dynamic_table_new_name [id=9cbc24e726d34722b5ff535376da654f]) successfully deleted
I20240503 02:13:12.591012 13038 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:12.594269 13168 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:12.594401 12908 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:12.595021 12975 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db2da39193164d1ebf830282258d47d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:12.595150 15653 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:12.596019 15653 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:12.598883 15643 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.615363 15651 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:12.616936 12908 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:12.620551 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:12.629125 13168 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:12.638428 9943 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:12.658936 15437 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.660652 15437 consensus_queue.cc:1035] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.683799 10074 tablet_service.cc:1467] Processing CreateTablet for tablet c0c131362451447583a346e17ea990e2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]), partition=
I20240503 02:13:12.684402 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c131362451447583a346e17ea990e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.684609 13153 tablet_service.cc:1467] Processing CreateTablet for tablet c0c131362451447583a346e17ea990e2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]), partition=
I20240503 02:13:12.685169 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c131362451447583a346e17ea990e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.688977 10469 tablet_service.cc:1467] Processing CreateTablet for tablet c0c131362451447583a346e17ea990e2 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]), partition=
I20240503 02:13:12.689584 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c0c131362451447583a346e17ea990e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.704881 15437 consensus_queue.cc:1035] T fde14f2778bc4b7f830bb7bdcbb6afd1 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.722169 15805 tablet_bootstrap.cc:492] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:12.725592 15805 tablet_bootstrap.cc:654] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.737223 15806 tablet_bootstrap.cc:492] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:12.737747 15805 tablet_bootstrap.cc:492] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:12.737979 15805 ts_tablet_manager.cc:1367] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20240503 02:13:12.739064 15805 raft_consensus.cc:348] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.739323 15805 raft_consensus.cc:374] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.739434 15805 raft_consensus.cc:729] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.739737 15805 consensus_queue.cc:260] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.750861 15805 ts_tablet_manager.cc:1398] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20240503 02:13:12.753602 15806 tablet_bootstrap.cc:654] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.761022 15807 tablet_bootstrap.cc:492] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:12.763713 15810 ts_tablet_manager.cc:902] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:12.765969 15806 tablet_bootstrap.cc:492] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:12.768987 15806 ts_tablet_manager.cc:1367] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.000s
I20240503 02:13:12.770062 15806 raft_consensus.cc:348] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.770454 15806 raft_consensus.cc:374] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.770586 15806 raft_consensus.cc:729] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.770984 15806 consensus_queue.cc:260] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.771807 15806 ts_tablet_manager.cc:1398] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:12.774443 10354 consensus_queue.cc:237] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:12.777267 15807 tablet_bootstrap.cc:654] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.777643 15811 raft_consensus.cc:482] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:12.777954 15811 raft_consensus.cc:504] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.779640 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c131362451447583a346e17ea990e2" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:12.779970 10091 raft_consensus.cc:2454] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:12.780802 13109 leader_election.cc:302] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:12.781174 15059 raft_consensus.cc:2781] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:12.781283 15059 raft_consensus.cc:482] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:12.781406 15059 raft_consensus.cc:3037] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.784642 15059 raft_consensus.cc:504] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.786144 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c131362451447583a346e17ea990e2" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:12.786419 10091 raft_consensus.cc:3037] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:12.791075 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c131362451447583a346e17ea990e2" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
W20240503 02:13:12.791882 13107 leader_election.cc:341] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:12.792265 10091 raft_consensus.cc:2454] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:12.794546 15817 ts_tablet_manager.cc:902] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:12.794912 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c131362451447583a346e17ea990e2" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
W20240503 02:13:12.795589 13107 leader_election.cc:341] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:12.796034 15059 leader_election.cc:288] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:12.796640 13109 leader_election.cc:302] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters: c6ec2f1283994a3089741715f811c850
I20240503 02:13:12.796813 15811 leader_election.cc:288] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:12.797622 15811 raft_consensus.cc:2781] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:12.799108 15811 raft_consensus.cc:686] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:13:12.800132 15811 consensus_queue.cc:237] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.808660 15709 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:12.810015 10295 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ee635c5c4d2c4befa0283cb8226f28d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:12.811391 10091 raft_consensus.cc:1261] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:12.812016 10224 raft_consensus.cc:1261] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:12.812434 15744 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:12.813228 15613 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:12.815765 9947 catalog_manager.cc:5484] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 1, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:12.830018 15711 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:12.835194 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_36"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:12.853595 15810 tablet_copy_client.cc:308] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:12.855930 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 674ea54f2cbd4c438cab5a7723c5a2c6 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:52743)
I20240503 02:13:12.856231 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 674ea54f2cbd4c438cab5a7723c5a2c6 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:52743: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-674ea54f2cbd4c438cab5a7723c5a2c6
I20240503 02:13:12.859669 10248 tablet_copy_source_session.cc:205] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:12.861676 15810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 674ea54f2cbd4c438cab5a7723c5a2c6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.868290 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2dbd2bd7014203bbd466201bf69e97 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.868997 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2dbd2bd7014203bbd466201bf69e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.873474 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2dbd2bd7014203bbd466201bf69e97 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.874060 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2dbd2bd7014203bbd466201bf69e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.875046 13290 tablet_service.cc:1467] Processing CreateTablet for tablet f38186393ba043a19f834d1d8780053a (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.875633 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38186393ba043a19f834d1d8780053a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.877909 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 8f2dbd2bd7014203bbd466201bf69e97 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.878552 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f2dbd2bd7014203bbd466201bf69e97. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.878861 13288 tablet_service.cc:1467] Processing CreateTablet for tablet 0c183608a78b4313b844ae4498d95955 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.879417 13288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c183608a78b4313b844ae4498d95955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.882077 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 0c183608a78b4313b844ae4498d95955 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.882654 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c183608a78b4313b844ae4498d95955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.886015 10469 tablet_service.cc:1467] Processing CreateTablet for tablet f38186393ba043a19f834d1d8780053a (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.886314 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 0c183608a78b4313b844ae4498d95955 (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.886592 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38186393ba043a19f834d1d8780053a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.886773 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0c183608a78b4313b844ae4498d95955. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.890502 10207 tablet_service.cc:1467] Processing CreateTablet for tablet f38186393ba043a19f834d1d8780053a (DEFAULT_TABLE table=rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:12.890978 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38186393ba043a19f834d1d8780053a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:12.899997 12904 raft_consensus.cc:1203] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 02:13:12.918581 15818 log.cc:924] Time spent T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Append to log took a long time: real 0.108s user 0.000s sys 0.000s
I20240503 02:13:12.921003 15613 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:12.922012 15827 ts_tablet_manager.cc:902] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:12.929301 15828 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:12.934689 9946 catalog_manager.cc:5484] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:12.945504 15807 tablet_bootstrap.cc:492] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:12.946507 15807 ts_tablet_manager.cc:1367] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.186s user 0.026s sys 0.013s
I20240503 02:13:12.947505 15828 tablet_bootstrap.cc:654] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.948379 15807 raft_consensus.cc:348] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.950676 15807 raft_consensus.cc:374] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.950927 15807 raft_consensus.cc:729] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.951306 15807 consensus_queue.cc:260] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:12.954444 15807 ts_tablet_manager.cc:1398] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:12.956480 15807 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:12.957734 15828 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:12.959761 15807 tablet_bootstrap.cc:654] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.960848 15837 tablet_bootstrap.cc:492] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:12.963057 15837 tablet_bootstrap.cc:654] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.964151 15828 ts_tablet_manager.cc:1367] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.000s
I20240503 02:13:12.965813 15828 raft_consensus.cc:348] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.967716 15828 raft_consensus.cc:374] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.967860 15828 raft_consensus.cc:729] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.968145 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:12.968765 15829 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:12.969411 15805 tablet_bootstrap.cc:492] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:12.971028 15829 tablet_bootstrap.cc:654] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.971467 15805 tablet_bootstrap.cc:654] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.972086 15810 tablet_copy_client.cc:783] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:12.972738 15810 tablet_copy_client.cc:647] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:12.975661 15828 consensus_queue.cc:260] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.976508 15828 ts_tablet_manager.cc:1398] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:13:12.976953 15828 tablet_bootstrap.cc:492] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:12.978582 15828 tablet_bootstrap.cc:654] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:12.987167 15829 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:12.987488 15829 ts_tablet_manager.cc:1367] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20240503 02:13:12.988525 15829 raft_consensus.cc:348] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.988866 15829 raft_consensus.cc:374] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:12.989002 15829 raft_consensus.cc:729] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:12.989372 15829 consensus_queue.cc:260] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.991335 9947 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]). Expected version 1 got 0
I20240503 02:13:12.991946 15829 ts_tablet_manager.cc:1398] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:12.992693 15829 tablet_bootstrap.cc:492] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:12.997258 10094 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:12.998435 15807 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:12.998598 15807 ts_tablet_manager.cc:1367] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20240503 02:13:13.000110 15807 raft_consensus.cc:348] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:12.986513 10227 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:13.002144 15807 raft_consensus.cc:374] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.002242 15807 raft_consensus.cc:729] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.002476 15807 consensus_queue.cc:260] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:13.004866 15807 ts_tablet_manager.cc:1398] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20240503 02:13:13.009227 15829 tablet_bootstrap.cc:654] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.012243 10491 raft_consensus.cc:3037] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.012269 10091 raft_consensus.cc:1261] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:13.013729 15752 consensus_queue.cc:1035] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.022171 10491 raft_consensus.cc:1261] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:13.023950 15829 tablet_bootstrap.cc:492] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:13.025233 15829 ts_tablet_manager.cc:1367] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.033s user 0.006s sys 0.001s
I20240503 02:13:13.028816 15829 raft_consensus.cc:348] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.033183 15829 raft_consensus.cc:374] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.034479 15829 raft_consensus.cc:729] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.035868 15829 consensus_queue.cc:260] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.044420 15829 ts_tablet_manager.cc:1398] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20240503 02:13:13.047931 9950 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]). Expected version 1 got 0
I20240503 02:13:13.050666 15834 tablet_bootstrap.cc:492] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:13.050670 15810 tablet_copy_client.cc:523] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:13.051458 15837 tablet_bootstrap.cc:492] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:13.051811 15837 ts_tablet_manager.cc:1367] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.014s
I20240503 02:13:13.053153 15837 raft_consensus.cc:348] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.053539 15837 raft_consensus.cc:374] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.053722 15837 raft_consensus.cc:729] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.054037 15837 consensus_queue.cc:260] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.055375 15827 tablet_copy_client.cc:308] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:13.056032 15810 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:13.056392 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet f1f1661a75a74746a4485b88b8e5b185 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:52779)
I20240503 02:13:13.056615 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet f1f1661a75a74746a4485b88b8e5b185 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:52779: session id = 47f901e617024baaa1865d12261ef84f-f1f1661a75a74746a4485b88b8e5b185
I20240503 02:13:13.056795 15837 ts_tablet_manager.cc:1398] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20240503 02:13:13.060871 12928 tablet_copy_source_session.cc:205] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.070801 9946 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]). Expected version 1 got 0
I20240503 02:13:13.083631 15805 tablet_bootstrap.cc:492] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:13.084514 15805 ts_tablet_manager.cc:1367] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.115s user 0.002s sys 0.011s
I20240503 02:13:13.085812 15805 raft_consensus.cc:348] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.086306 15805 raft_consensus.cc:374] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.086465 15805 raft_consensus.cc:729] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.087074 15805 consensus_queue.cc:260] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.088732 15827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f1f1661a75a74746a4485b88b8e5b185. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.108438 15856 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:13.115680 15856 tablet.cc:1620] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:13.117739 9946 catalog_manager.cc:5576] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) does not have the latest schema for tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]). Expected version 1 got 0
I20240503 02:13:13.128005 9947 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]). Expected version 1 got 0
I20240503 02:13:13.131264 15828 tablet_bootstrap.cc:492] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:13.131558 15828 ts_tablet_manager.cc:1367] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.155s user 0.032s sys 0.053s
I20240503 02:13:13.132416 15828 raft_consensus.cc:348] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.132802 15828 raft_consensus.cc:374] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.132908 15828 raft_consensus.cc:729] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.133241 15828 consensus_queue.cc:260] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.134073 15828 ts_tablet_manager.cc:1398] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:13.132377 15854 ts_tablet_manager.cc:902] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:13.139864 15437 raft_consensus.cc:482] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.140327 15437 raft_consensus.cc:504] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:13.143447 15437 leader_election.cc:288] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:13.149529 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38186393ba043a19f834d1d8780053a" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:13.149922 10224 raft_consensus.cc:2454] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:13.152709 15248 raft_consensus.cc:482] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.153025 15248 raft_consensus.cc:504] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:13.154600 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c131362451447583a346e17ea990e2" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:13.155723 15248 leader_election.cc:288] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:13.156317 13245 leader_election.cc:302] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:13.158480 15643 raft_consensus.cc:2781] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:13.158617 15643 raft_consensus.cc:482] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:13.158720 15643 raft_consensus.cc:3037] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.160281 15846 tablet_bootstrap.cc:492] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:13.162040 15643 raft_consensus.cc:504] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:13.162676 15846 tablet_bootstrap.cc:654] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.165174 15643 leader_election.cc:288] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:13.166442 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38186393ba043a19f834d1d8780053a" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:13.166746 10489 raft_consensus.cc:2454] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:13.167076 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38186393ba043a19f834d1d8780053a" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:13.167229 10490 raft_consensus.cc:3037] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.110255 15855 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:13.169191 15855 tablet.cc:1620] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:13.175034 15805 ts_tablet_manager.cc:1398] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.090s user 0.000s sys 0.002s
I20240503 02:13:13.180609 15834 tablet_bootstrap.cc:654] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.186802 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f38186393ba043a19f834d1d8780053a" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:13.187259 10227 raft_consensus.cc:3037] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.190029 10490 raft_consensus.cc:2454] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:13.190681 9943 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b] alter complete (version 1)
I20240503 02:13:13.191747 15740 raft_consensus.cc:482] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.191962 15740 raft_consensus.cc:504] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.124245 15059 consensus_queue.cc:1035] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.166952 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c0c131362451447583a346e17ea990e2" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:13.195485 15740 leader_election.cc:288] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:13.196271 13303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c183608a78b4313b844ae4498d95955" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:13.196584 13303 raft_consensus.cc:2454] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:13.198626 10159 leader_election.cc:302] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:13.199016 15740 raft_consensus.cc:2781] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:13.199105 15740 raft_consensus.cc:482] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:13.199216 15740 raft_consensus.cc:3037] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.203469 15740 raft_consensus.cc:504] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.204797 13303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c183608a78b4313b844ae4498d95955" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:13.205096 13303 raft_consensus.cc:3037] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.209054 13303 raft_consensus.cc:2454] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:13.209689 10159 leader_election.cc:302] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:13.210058 15629 raft_consensus.cc:2781] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:13.210240 15629 raft_consensus.cc:686] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:13.210662 15629 consensus_queue.cc:237] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.221933 15740 leader_election.cc:288] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:13.225802 13243 leader_election.cc:302] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:13.226650 15437 raft_consensus.cc:2781] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:13.226698 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c183608a78b4313b844ae4498d95955" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:13:13.226847 15437 raft_consensus.cc:686] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:13.227246 15437 consensus_queue.cc:237] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:13.228565 10357 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c183608a78b4313b844ae4498d95955" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:13.230873 15846 tablet_bootstrap.cc:492] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:13.235311 15834 tablet_bootstrap.cc:492] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:13.235792 15834 ts_tablet_manager.cc:1367] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.185s user 0.006s sys 0.000s
W20240503 02:13:13.242084 10161 leader_election.cc:341] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Illegal state: must be running to vote when last-logged opid is not known
W20240503 02:13:13.243180 10161 leader_election.cc:341] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:13.243791 15846 ts_tablet_manager.cc:1367] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.084s user 0.006s sys 0.006s
I20240503 02:13:13.241719 15866 ts_tablet_manager.cc:902] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:13.248415 15846 raft_consensus.cc:348] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.250581 15846 raft_consensus.cc:374] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.251120 15846 raft_consensus.cc:729] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.253211 15846 consensus_queue.cc:260] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.255352 9951 catalog_manager.cc:5484] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:13.256469 15846 ts_tablet_manager.cc:1398] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20240503 02:13:13.232411 10427 leader_election.cc:302] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ec2f1283994a3089741715f811c850; no voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad
W20240503 02:13:13.281292 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 48943.6us
I20240503 02:13:13.288707 10227 raft_consensus.cc:2454] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:13.295193 9943 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:13.297535 9951 catalog_manager.cc:5484] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:13.308274 15810 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:13.308907 15810 tablet_bootstrap.cc:492] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:13.309270 15810 ts_tablet_manager.cc:1367] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.253s user 0.053s sys 0.012s
I20240503 02:13:13.310389 15810 raft_consensus.cc:348] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:13.310750 15810 raft_consensus.cc:729] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:13.311283 15856 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:13.311893 15856 tablet.cc:1620] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:13.314412 15855 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:13.315128 15855 tablet.cc:1620] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: MemRowSet was empty: no flush needed.
I20240503 02:13:13.311010 15810 consensus_queue.cc:260] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:13.320675 15810 ts_tablet_manager.cc:1398] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:13:13.321607 10248 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-674ea54f2cbd4c438cab5a7723c5a2c6 received from {username='slave'} at 127.4.5.72:52743
I20240503 02:13:13.321785 10248 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-674ea54f2cbd4c438cab5a7723c5a2c6 on tablet 674ea54f2cbd4c438cab5a7723c5a2c6 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:13.322252 13303 raft_consensus.cc:1203] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
W20240503 02:13:13.331763 15867 log.cc:924] Time spent T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082: Append to log took a long time: real 0.095s user 0.000s sys 0.000s
W20240503 02:13:13.325237 12976 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 50729.2us
W20240503 02:13:13.337424 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 54885.1us
I20240503 02:13:13.337265 9950 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b] alter complete (version 2)
I20240503 02:13:13.338495 15834 raft_consensus.cc:348] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.343003 15854 tablet_copy_client.cc:308] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:13.344193 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 13e83ce8820b491992c041a8c21fa3df from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:59439)
I20240503 02:13:13.344377 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 13e83ce8820b491992c041a8c21fa3df from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:59439: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-13e83ce8820b491992c041a8c21fa3df
I20240503 02:13:13.346534 15834 raft_consensus.cc:374] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.346719 15834 raft_consensus.cc:729] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.347378 10248 tablet_copy_source_session.cc:205] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.347110 15834 consensus_queue.cc:260] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.352623 15879 ts_tablet_manager.cc:902] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:13.353389 15817 tablet_copy_client.cc:308] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:13.354315 15834 ts_tablet_manager.cc:1398] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
I20240503 02:13:13.354285 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:60931)
I20240503 02:13:13.354490 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:60931: session id = f93b103af083471b8e8c0376e9148aad-2fadc7e9c7604b5d8f7b8751b6ca78a7
I20240503 02:13:13.359774 15827 tablet_copy_client.cc:783] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.360283 15827 tablet_copy_client.cc:647] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.364394 10115 tablet_copy_source_session.cc:205] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.365321 15827 tablet_copy_client.cc:523] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:13.365984 15854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e83ce8820b491992c041a8c21fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.366681 15817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.383020 15817 tablet_copy_client.cc:783] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.383644 15817 tablet_copy_client.cc:647] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.383700 15854 tablet_copy_client.cc:783] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.384016 15854 tablet_copy_client.cc:647] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.386385 15817 tablet_copy_client.cc:523] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:13.386474 15854 tablet_copy_client.cc:523] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:13.390779 15854 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:13.412882 15879 tablet_copy_client.cc:308] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:13.414211 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet ee635c5c4d2c4befa0283cb8226f28d7 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:49771)
I20240503 02:13:13.414490 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet ee635c5c4d2c4befa0283cb8226f28d7 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:49771: session id = f93b103af083471b8e8c0376e9148aad-ee635c5c4d2c4befa0283cb8226f28d7
I20240503 02:13:13.417291 15889 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:13.417595 10380 tablet_copy_source_session.cc:205] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.418509 15248 raft_consensus.cc:2726] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:13.422613 15879 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee635c5c4d2c4befa0283cb8226f28d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.422667 9947 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]). Expected version 2 got 0
I20240503 02:13:13.427150 15866 tablet_copy_client.cc:308] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:13.429188 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet db2da39193164d1ebf830282258d47d7 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:42943)
I20240503 02:13:13.429464 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet db2da39193164d1ebf830282258d47d7 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:42943: session id = 47f901e617024baaa1865d12261ef84f-db2da39193164d1ebf830282258d47d7
I20240503 02:13:13.430361 15817 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:13.431468 15889 tablet.cc:1620] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:13:13.432049 15827 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:13.443589 13061 tablet_copy_source_session.cc:205] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:13.445108 15703 raft_consensus.cc:1050] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:13.446177 15703 consensus_queue.cc:237] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:13.450501 9943 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table [id=df28d4ef8f024fb0aa0e9ce37b29082b]). Expected version 2 got 1
I20240503 02:13:13.455236 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:13.466648 15889 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:13.466936 15855 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:13.467231 15889 tablet.cc:1620] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:13:13.471233 15856 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:13.472597 12904 raft_consensus.cc:1261] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:13.473322 15878 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:13.475123 15879 tablet_copy_client.cc:783] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.475503 15879 tablet_copy_client.cc:647] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.487752 15827 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:13.488530 15879 tablet_copy_client.cc:523] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:13.488370 15827 tablet_bootstrap.cc:492] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:13.493798 15827 ts_tablet_manager.cc:1367] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.000s
I20240503 02:13:13.495473 15827 raft_consensus.cc:348] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:13.495846 15827 raft_consensus.cc:729] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:13.496137 15827 consensus_queue.cc:260] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:13.497704 15879 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:13.500777 9950 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=df28d4ef8f024fb0aa0e9ce37b29082b] alter complete (version 3)
I20240503 02:13:13.502985 15827 ts_tablet_manager.cc:1398] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20240503 02:13:13.503999 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-f1f1661a75a74746a4485b88b8e5b185 received from {username='slave'} at 127.4.5.67:52779
I20240503 02:13:13.504228 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session 47f901e617024baaa1865d12261ef84f-f1f1661a75a74746a4485b88b8e5b185 on tablet f1f1661a75a74746a4485b88b8e5b185 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:13.512471 13168 raft_consensus.cc:1261] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:13.512377 15709 raft_consensus.cc:482] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.512676 15709 raft_consensus.cc:504] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.513190 15878 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:13.513484 15709 leader_election.cc:288] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:13.514496 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2dbd2bd7014203bbd466201bf69e97" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:13.514757 13168 raft_consensus.cc:2454] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:13.515937 12843 leader_election.cc:302] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:13.516860 15709 raft_consensus.cc:2781] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:13.516819 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2dbd2bd7014203bbd466201bf69e97" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:13.516954 15709 raft_consensus.cc:482] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:13.517081 15709 raft_consensus.cc:3037] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.517107 13039 raft_consensus.cc:2454] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:13.520622 13305 raft_consensus.cc:1261] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:13.521284 15699 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:13.522723 15709 raft_consensus.cc:504] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.523679 15709 leader_election.cc:288] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:13.524163 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2dbd2bd7014203bbd466201bf69e97" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:13.524420 13039 raft_consensus.cc:3037] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.527689 13039 raft_consensus.cc:2454] T 8f2dbd2bd7014203bbd466201bf69e97 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:13.529206 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f2dbd2bd7014203bbd466201bf69e97" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:13.529517 13168 raft_consensus.cc:3037] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.534502 12843 leader_election.cc:302] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:13.537464 15715 raft_consensus.cc:2781] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:13.537701 15715 raft_consensus.cc:686] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:13.538085 15715 consensus_queue.cc:237] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:13.542212 15699 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:13.542680 13168 raft_consensus.cc:2454] T 8f2dbd2bd7014203bbd466201bf69e97 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:13.549855 13305 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:13.561554 13172 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:13.567298 9951 catalog_manager.cc:5484] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:13.588685 15889 tablet.cc:1722] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:13.589221 9947 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table_new_name [id=df28d4ef8f024fb0aa0e9ce37b29082b]). Expected version 3 got 1
I20240503 02:13:13.592909 9950 catalog_manager.cc:5484] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:13.594316 15854 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:13.594908 15854 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:13.595264 15854 ts_tablet_manager.cc:1367] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.205s user 0.051s sys 0.014s
I20240503 02:13:13.596271 15854 raft_consensus.cc:348] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:13.596626 15854 raft_consensus.cc:729] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:13.596871 15854 consensus_queue.cc:260] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:13.597779 15854 ts_tablet_manager.cc:1398] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:13.598665 10248 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-13e83ce8820b491992c041a8c21fa3df received from {username='slave'} at 127.4.5.69:59439
I20240503 02:13:13.598805 10248 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-13e83ce8820b491992c041a8c21fa3df on tablet 13e83ce8820b491992c041a8c21fa3df with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:13.607762 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:13.608037 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:13.608299 15866 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2da39193164d1ebf830282258d47d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.616420 15437 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.619331 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet c0c131362451447583a346e17ea990e2
I20240503 02:13:13.620190 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c131362451447583a346e17ea990e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:13 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:13.620687 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c131362451447583a346e17ea990e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:13 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:13.621098 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet c0c131362451447583a346e17ea990e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:13 UTC) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:13.627594 15744 raft_consensus.cc:482] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:13.627846 15744 raft_consensus.cc:504] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:13.628645 15744 leader_election.cc:288] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:13.646751 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c183608a78b4313b844ae4498d95955" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:13.647166 13305 raft_consensus.cc:2379] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 47f901e617024baaa1865d12261ef84f in current term 1: Already voted for candidate eff8e8d11a4447218741bb994aa41082 in this term.
I20240503 02:13:13.668503 15643 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.675256 15740 consensus_queue.cc:1035] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.675971 15915 tablet_replica.cc:330] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:13.677366 15915 raft_consensus.cc:2229] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:13.677848 15915 raft_consensus.cc:2258] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.681900 15915 ts_tablet_manager.cc:1875] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:13.685882 15917 tablet_replica.cc:330] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:13.686211 15917 raft_consensus.cc:2229] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:13.686760 15917 raft_consensus.cc:2258] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.690155 10354 raft_consensus.cc:3037] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:13.692435 12907 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:13.696161 15629 consensus_queue.cc:1035] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:13.700116 15915 ts_tablet_manager.cc:1888] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:13.717653 15915 log.cc:1196] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/c0c131362451447583a346e17ea990e2
I20240503 02:13:13.718880 15915 ts_tablet_manager.cc:1909] T c0c131362451447583a346e17ea990e2 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:13:13.722126 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table_new_name [id=df28d4ef8f024fb0aa0e9ce37b29082b]) successfully deleted
I20240503 02:13:13.730675 15916 tablet_replica.cc:330] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:13.731139 15916 raft_consensus.cc:2229] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:13.731487 15916 raft_consensus.cc:2258] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.733481 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:13.747100 10089 consensus_queue.cc:237] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:13.748392 15916 ts_tablet_manager.cc:1875] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:13.751138 15917 ts_tablet_manager.cc:1875] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:13.751720 13305 raft_consensus.cc:1261] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:13.752533 15703 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:13.759303 15703 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:13.760110 12904 raft_consensus.cc:1261] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:13.766238 13305 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:13.766830 10224 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "0c183608a78b4313b844ae4498d95955" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:13.768501 10294 leader_election.cc:302] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47f901e617024baaa1865d12261ef84f; no voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082
I20240503 02:13:13.768999 15613 raft_consensus.cc:2726] T 0c183608a78b4313b844ae4498d95955 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:13.774086 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 432c8c5d42cb454fb7d1ca5af4782bb7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:13.775424 9950 catalog_manager.cc:5484] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:13.777411 10357 raft_consensus.cc:1203] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:13.777851 15699 consensus_queue.cc:1035] T 432c8c5d42cb454fb7d1ca5af4782bb7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:13.780347 12904 raft_consensus.cc:2932] T 432c8c5d42cb454fb7d1ca5af4782bb7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:13.795698 15879 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:13.796298 15879 tablet_bootstrap.cc:492] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:13.808171 15879 ts_tablet_manager.cc:1367] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.310s user 0.057s sys 0.006s
I20240503 02:13:13.809599 15879 raft_consensus.cc:348] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:13.809958 15879 raft_consensus.cc:729] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:13.810294 15879 consensus_queue.cc:260] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:13.814682 15879 ts_tablet_manager.cc:1398] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:13.816004 10337 tablet_service.cc:1467] Processing CreateTablet for tablet e1784f1f151e4617aaff8ed526ecd316 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30]), partition=
I20240503 02:13:13.816169 12887 tablet_service.cc:1467] Processing CreateTablet for tablet e1784f1f151e4617aaff8ed526ecd316 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30]), partition=
I20240503 02:13:13.816567 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1784f1f151e4617aaff8ed526ecd316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.816643 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1784f1f151e4617aaff8ed526ecd316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.816890 10074 tablet_service.cc:1467] Processing CreateTablet for tablet e1784f1f151e4617aaff8ed526ecd316 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30]), partition=
I20240503 02:13:13.817348 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e1784f1f151e4617aaff8ed526ecd316. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:13.820626 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-ee635c5c4d2c4befa0283cb8226f28d7 received from {username='slave'} at 127.4.5.71:49771
I20240503 02:13:13.820787 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session f93b103af083471b8e8c0376e9148aad-ee635c5c4d2c4befa0283cb8226f28d7 on tablet ee635c5c4d2c4befa0283cb8226f28d7 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:13.836373 15926 tablet_bootstrap.cc:492] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:13.840904 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet 432c8c5d42cb454fb7d1ca5af4782bb7 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:13.843613 15926 tablet_bootstrap.cc:654] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.850257 15866 tablet_copy_client.cc:783] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:13.850703 15866 tablet_copy_client.cc:647] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:13.854605 15927 tablet_bootstrap.cc:492] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:13.856545 15866 tablet_copy_client.cc:523] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:13.857841 15927 tablet_bootstrap.cc:654] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.858870 15926 tablet_bootstrap.cc:492] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:13.860018 15926 ts_tablet_manager.cc:1367] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.003s
I20240503 02:13:13.860980 15926 raft_consensus.cc:348] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:13.861239 15926 raft_consensus.cc:374] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.861297 15926 raft_consensus.cc:729] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.864197 15916 ts_tablet_manager.cc:1888] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:13.869750 15926 consensus_queue.cc:260] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:13.871038 15866 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:13.871948 15740 raft_consensus.cc:1050] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:13.872706 15925 tablet_bootstrap.cc:492] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:13.873368 15740 consensus_queue.cc:237] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:13.875627 15916 log.cc:1196] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/c0c131362451447583a346e17ea990e2
I20240503 02:13:13.880621 13303 raft_consensus.cc:1261] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:13.880789 15817 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:13.881378 15817 tablet_bootstrap.cc:492] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:13.881783 15817 ts_tablet_manager.cc:1367] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.452s user 0.058s sys 0.006s
I20240503 02:13:13.882898 15817 raft_consensus.cc:348] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:13.883231 15817 raft_consensus.cc:729] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:13.883481 15817 consensus_queue.cc:260] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:13.884232 15817 ts_tablet_manager.cc:1398] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:13.871814 15926 ts_tablet_manager.cc:1398] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20240503 02:13:13.884709 15741 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:13.895046 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-2fadc7e9c7604b5d8f7b8751b6ca78a7 received from {username='slave'} at 127.4.5.71:60931
I20240503 02:13:13.895263 15916 ts_tablet_manager.cc:1909] T c0c131362451447583a346e17ea990e2 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:13:13.895284 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session f93b103af083471b8e8c0376e9148aad-2fadc7e9c7604b5d8f7b8751b6ca78a7 on tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:13.897279 15917 ts_tablet_manager.cc:1888] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:13.897508 15917 log.cc:1196] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/c0c131362451447583a346e17ea990e2
I20240503 02:13:13.898171 15917 ts_tablet_manager.cc:1909] T c0c131362451447583a346e17ea990e2 P f93b103af083471b8e8c0376e9148aad: Deleting consensus metadata
I20240503 02:13:13.905700 15925 tablet_bootstrap.cc:654] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:13.906589 10491 raft_consensus.cc:1261] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:13.907555 15741 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:13.911741 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table_new_name [id=df28d4ef8f024fb0aa0e9ce37b29082b]) successfully deleted
I20240503 02:13:13.914503 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet c0c131362451447583a346e17ea990e2 (table rebalancer_dynamic_table_new_name [id=df28d4ef8f024fb0aa0e9ce37b29082b]) successfully deleted
I20240503 02:13:13.916857 10357 raft_consensus.cc:1261] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:13.917809 15917 tablet_replica.cc:330] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:13.918310 15917 raft_consensus.cc:2229] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:13.918779 15917 raft_consensus.cc:2258] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:13.924749 15917 ts_tablet_manager.cc:1875] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:13.933069 15925 tablet_bootstrap.cc:492] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:13.933753 15925 ts_tablet_manager.cc:1367] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.061s user 0.005s sys 0.002s
I20240503 02:13:13.918785 15740 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.017s
I20240503 02:13:13.935366 15925 raft_consensus.cc:348] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:13.936801 15925 raft_consensus.cc:374] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:13.936945 15925 raft_consensus.cc:729] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:13.941226 15925 consensus_queue.cc:260] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:13.941680 15866 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:13.942178 15866 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:13.942494 15866 ts_tablet_manager.cc:1367] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.072s user 0.016s sys 0.030s
I20240503 02:13:13.942848 15925 ts_tablet_manager.cc:1398] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.009s user 0.002s sys 0.001s
I20240503 02:13:13.943521 15866 raft_consensus.cc:348] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:13.943826 15866 raft_consensus.cc:729] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:13.944056 15866 consensus_queue.cc:260] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:13.944932 15866 ts_tablet_manager.cc:1398] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:13.946995 15917 ts_tablet_manager.cc:1888] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:13.947245 15917 log.cc:1196] T 432c8c5d42cb454fb7d1ca5af4782bb7 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/432c8c5d42cb454fb7d1ca5af4782bb7
I20240503 02:13:13.947508 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-db2da39193164d1ebf830282258d47d7 received from {username='slave'} at 127.4.5.67:42943
I20240503 02:13:13.947734 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session 47f901e617024baaa1865d12261ef84f-db2da39193164d1ebf830282258d47d7 on tablet db2da39193164d1ebf830282258d47d7 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:13.948649 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 432c8c5d42cb454fb7d1ca5af4782bb7 (table rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]) successfully deleted
I20240503 02:13:13.966271 13168 raft_consensus.cc:1203] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:13.999521 12904 raft_consensus.cc:1203] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:14.007813 15709 consensus_queue.cc:1035] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.013864 15613 raft_consensus.cc:482] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.014146 15613 raft_consensus.cc:504] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:14.015178 15613 leader_election.cc:288] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:14.015820 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1784f1f151e4617aaff8ed526ecd316" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:14.015977 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1784f1f151e4617aaff8ed526ecd316" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:14.016137 12904 raft_consensus.cc:2454] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
W20240503 02:13:14.016775 10294 leader_election.cc:341] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.017128 10292 leader_election.cc:302] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:14.018729 15613 raft_consensus.cc:2781] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.018859 15613 raft_consensus.cc:482] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.018950 15613 raft_consensus.cc:3037] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.022855 15613 raft_consensus.cc:504] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:14.024679 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1784f1f151e4617aaff8ed526ecd316" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:14.024708 12904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e1784f1f151e4617aaff8ed526ecd316" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:14.024979 12904 raft_consensus.cc:3037] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:13:14.025496 10294 leader_election.cc:341] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.028357 12904 raft_consensus.cc:2454] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:14.031456 10292 leader_election.cc:302] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:14.031838 15744 raft_consensus.cc:2781] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:14.032238 15613 leader_election.cc:288] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:14.040064 15741 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:14.051477 15744 raft_consensus.cc:686] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:13:14.051967 15744 consensus_queue.cc:237] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:14.072204 15927 tablet_bootstrap.cc:492] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:14.072474 15927 ts_tablet_manager.cc:1367] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.218s user 0.047s sys 0.097s
I20240503 02:13:14.073887 15927 raft_consensus.cc:348] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:14.074185 15927 raft_consensus.cc:374] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.074282 15927 raft_consensus.cc:729] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.074613 15927 consensus_queue.cc:260] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:14.075325 15927 ts_tablet_manager.cc:1398] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:14.063407 13305 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:14.057093 10357 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:14.091764 15709 consensus_queue.cc:1035] T 8f2dbd2bd7014203bbd466201bf69e97 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.095111 10491 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:14.107086 9947 catalog_manager.cc:5484] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:14.123808 10224 consensus_queue.cc:237] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:14.126827 9951 catalog_manager.cc:5484] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:14.181536 10489 raft_consensus.cc:1261] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:14.190131 15741 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.005s
W20240503 02:13:14.191831 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 122216us
I20240503 02:13:14.200955 15629 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:14.204941 13302 raft_consensus.cc:1261] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:14.210306 15741 consensus_queue.cc:1035] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:14.212751 10491 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:14.213574 13302 raft_consensus.cc:2932] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:14.219033 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 674ea54f2cbd4c438cab5a7723c5a2c6 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:14.220957 9951 catalog_manager.cc:5484] T 674ea54f2cbd4c438cab5a7723c5a2c6 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:14.221309 9950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:14.229353 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 674ea54f2cbd4c438cab5a7723c5a2c6 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:14.231048 15954 tablet_replica.cc:330] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:14.231370 15954 raft_consensus.cc:2229] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:14.231662 15954 raft_consensus.cc:2258] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:14.234001 15954 ts_tablet_manager.cc:1875] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:14.239837 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 8710d52bcd6b45bdb4babf0ca987486c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.240417 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8710d52bcd6b45bdb4babf0ca987486c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.240602 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 818345467bd9400aa5fdd875b59f3249 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.241081 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818345467bd9400aa5fdd875b59f3249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.240873 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 8710d52bcd6b45bdb4babf0ca987486c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.245496 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8710d52bcd6b45bdb4babf0ca987486c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.250375 15715 raft_consensus.cc:1050] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:14.251020 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 8710d52bcd6b45bdb4babf0ca987486c (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.251590 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8710d52bcd6b45bdb4babf0ca987486c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.251504 15715 consensus_queue.cc:237] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:14.255824 13153 tablet_service.cc:1467] Processing CreateTablet for tablet 818345467bd9400aa5fdd875b59f3249 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.256399 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818345467bd9400aa5fdd875b59f3249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.259526 10471 tablet_service.cc:1467] Processing CreateTablet for tablet 818345467bd9400aa5fdd875b59f3249 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.260023 10471 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818345467bd9400aa5fdd875b59f3249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.262776 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 004a7ce056044d25ab68a44e51688ca3 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.263324 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004a7ce056044d25ab68a44e51688ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.264883 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 004a7ce056044d25ab68a44e51688ca3 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.265296 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004a7ce056044d25ab68a44e51688ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.270767 9950 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:14.273731 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 004a7ce056044d25ab68a44e51688ca3 (DEFAULT_TABLE table=rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:14.274361 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004a7ce056044d25ab68a44e51688ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:14.274785 15957 tablet_bootstrap.cc:492] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:14.276824 15957 tablet_bootstrap.cc:654] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.280651 10089 raft_consensus.cc:1261] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:14.281692 10357 raft_consensus.cc:1261] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:14.282509 15711 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:14.283355 15711 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:14.293093 15925 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:14.295235 15925 tablet_bootstrap.cc:654] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.310348 15927 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:14.312835 15927 tablet_bootstrap.cc:654] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.327292 10491 raft_consensus.cc:1261] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:14.329028 15711 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:14.329638 15927 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:14.331943 15927 ts_tablet_manager.cc:1367] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.008s
I20240503 02:13:14.333088 15927 raft_consensus.cc:348] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.333329 15927 raft_consensus.cc:374] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.333410 15927 raft_consensus.cc:729] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.333688 15927 consensus_queue.cc:260] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.334538 15927 ts_tablet_manager.cc:1398] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:14.346629 15925 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:14.347060 15925 ts_tablet_manager.cc:1367] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.054s user 0.009s sys 0.004s
I20240503 02:13:14.348165 15925 raft_consensus.cc:348] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.348621 15925 raft_consensus.cc:374] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.348793 15925 raft_consensus.cc:729] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.349375 10094 raft_consensus.cc:3037] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.350867 15956 tablet_bootstrap.cc:492] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:14.352723 15956 tablet_bootstrap.cc:654] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.353308 12904 raft_consensus.cc:1261] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:14.356312 10094 raft_consensus.cc:1261] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:14.357216 15925 consensus_queue.cc:260] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.357625 15613 consensus_queue.cc:1035] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.357890 15744 consensus_queue.cc:1035] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:14.359633 10359 raft_consensus.cc:1203] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:14.365200 15925 ts_tablet_manager.cc:1398] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20240503 02:13:14.376140 15954 ts_tablet_manager.cc:1888] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:14.377426 13168 raft_consensus.cc:1203] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:14.382151 15957 tablet_bootstrap.cc:492] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:14.383716 15957 ts_tablet_manager.cc:1367] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.109s user 0.017s sys 0.014s
I20240503 02:13:14.383160 15709 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.389446 15954 log.cc:1196] T 674ea54f2cbd4c438cab5a7723c5a2c6 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/674ea54f2cbd4c438cab5a7723c5a2c6
I20240503 02:13:14.389765 15957 raft_consensus.cc:348] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.391532 15957 raft_consensus.cc:374] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.392016 15957 raft_consensus.cc:729] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.392927 15957 consensus_queue.cc:260] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.393872 15977 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:14.394483 15977 tablet.cc:1620] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: MemRowSet was empty: no flush needed.
I20240503 02:13:14.394495 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 674ea54f2cbd4c438cab5a7723c5a2c6 (table rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]) successfully deleted
I20240503 02:13:14.395323 15957 ts_tablet_manager.cc:1398] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20240503 02:13:14.415032 15976 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:14.417636 15976 tablet_bootstrap.cc:654] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.418449 15956 tablet_bootstrap.cc:492] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:14.418845 15956 ts_tablet_manager.cc:1367] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.000s
I20240503 02:13:14.420011 15956 raft_consensus.cc:348] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.420562 15956 raft_consensus.cc:374] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.420688 15956 raft_consensus.cc:729] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.421154 15956 consensus_queue.cc:260] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.422778 15956 ts_tablet_manager.cc:1398] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:14.436599 15956 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:14.427636 10089 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.452662 9947 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet e1784f1f151e4617aaff8ed526ecd316 (table rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30]). Expected version 1 got 0
I20240503 02:13:14.453523 15956 tablet_bootstrap.cc:654] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.459421 15437 raft_consensus.cc:482] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.459708 15437 raft_consensus.cc:504] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.461338 15437 leader_election.cc:288] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:14.461596 15699 raft_consensus.cc:1050] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:14.462486 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8710d52bcd6b45bdb4babf0ca987486c" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:14.462858 10491 raft_consensus.cc:2454] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:14.462909 15699 consensus_queue.cc:237] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:14.463795 13243 leader_election.cc:302] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:14.464180 15437 raft_consensus.cc:2781] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.464311 15437 raft_consensus.cc:482] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.464408 15437 raft_consensus.cc:3037] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.475793 15437 raft_consensus.cc:504] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.482064 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8710d52bcd6b45bdb4babf0ca987486c" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
W20240503 02:13:14.483012 13246 leader_election.cc:341] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.485448 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8710d52bcd6b45bdb4babf0ca987486c" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:14.485706 10491 raft_consensus.cc:3037] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.488977 10491 raft_consensus.cc:2454] T 8710d52bcd6b45bdb4babf0ca987486c P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:14.489193 13172 raft_consensus.cc:1261] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.489694 13243 leader_election.cc:302] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:14.490139 15643 raft_consensus.cc:2781] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:14.490757 15703 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.502839 15976 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:14.503087 15976 ts_tablet_manager.cc:1367] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.088s user 0.003s sys 0.023s
I20240503 02:13:14.504072 15976 raft_consensus.cc:348] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.504341 15976 raft_consensus.cc:374] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.504472 15976 raft_consensus.cc:729] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.504726 15976 consensus_queue.cc:260] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.507122 10354 raft_consensus.cc:1261] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.507872 15703 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.515983 15976 ts_tablet_manager.cc:1398] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20240503 02:13:14.516427 15976 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:14.518388 15976 tablet_bootstrap.cc:654] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.521147 15956 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:14.521363 15956 ts_tablet_manager.cc:1367] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.089s user 0.003s sys 0.003s
I20240503 02:13:14.533437 10224 raft_consensus.cc:1261] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.534653 15976 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:14.534617 15703 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.535362 15976 ts_tablet_manager.cc:1367] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.003s
I20240503 02:13:14.535485 15703 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.536314 15976 raft_consensus.cc:348] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.536602 15976 raft_consensus.cc:374] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.536684 15976 raft_consensus.cc:729] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.538623 15976 consensus_queue.cc:260] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.539811 15976 ts_tablet_manager.cc:1398] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:14.541390 10355 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.557152 13038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8710d52bcd6b45bdb4babf0ca987486c" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:14.555963 10224 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
W20240503 02:13:14.560814 13246 leader_election.cc:341] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.567667 13172 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.578262 9947 catalog_manager.cc:5484] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.580853 15956 raft_consensus.cc:348] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.588183 15956 raft_consensus.cc:374] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.591262 15956 raft_consensus.cc:729] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.594801 15956 consensus_queue.cc:260] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.597981 15956 ts_tablet_manager.cc:1398] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.076s user 0.000s sys 0.003s
I20240503 02:13:14.599680 15437 leader_election.cc:288] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:14.605602 15956 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:14.611225 15703 raft_consensus.cc:482] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.611716 15703 raft_consensus.cc:504] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.614521 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818345467bd9400aa5fdd875b59f3249" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:14.614873 13168 raft_consensus.cc:2454] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:14.615592 10029 leader_election.cc:302] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:14.615985 15878 raft_consensus.cc:2781] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.616070 15878 raft_consensus.cc:482] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.616153 15878 raft_consensus.cc:3037] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.617231 15980 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:14.619164 15878 raft_consensus.cc:504] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.619789 15980 tablet.cc:1620] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:14.620772 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818345467bd9400aa5fdd875b59f3249" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:14.621127 13168 raft_consensus.cc:3037] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.623035 15740 raft_consensus.cc:1050] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:14.624241 15740 consensus_queue.cc:237] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:14.625017 15878 leader_election.cc:288] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:14.626829 13168 raft_consensus.cc:2454] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:14.627542 10029 leader_election.cc:302] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:14.628129 15878 raft_consensus.cc:2781] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:14.634621 15878 raft_consensus.cc:686] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:13:14.636088 12906 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.636970 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818345467bd9400aa5fdd875b59f3249" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:14.636940 15629 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.637215 10489 raft_consensus.cc:3037] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.641671 10489 raft_consensus.cc:2454] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:14.646580 9943 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet e1784f1f151e4617aaff8ed526ecd316 (table rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30]). Expected version 1 got 0
I20240503 02:13:14.648329 10359 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.651177 15741 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.651007 10489 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.653403 15811 raft_consensus.cc:482] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.653635 15811 raft_consensus.cc:504] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.654776 15811 leader_election.cc:288] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:14.655129 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
W20240503 02:13:14.656090 13107 leader_election.cc:341] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.663254 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:14.663616 12908 raft_consensus.cc:2454] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 0.
I20240503 02:13:14.663786 15711 raft_consensus.cc:482] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:14.663944 15711 raft_consensus.cc:504] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.664482 15878 consensus_queue.cc:237] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:14.664822 15711 leader_election.cc:288] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:14.666889 13167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:14.667213 13167 raft_consensus.cc:2454] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:14.667907 12843 leader_election.cc:302] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:14.668777 15715 raft_consensus.cc:2781] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.669538 15715 raft_consensus.cc:482] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.669673 15715 raft_consensus.cc:3037] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:14.672698 15956 tablet_bootstrap.cc:654] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.674343 13107 leader_election.cc:302] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters: c6ec2f1283994a3089741715f811c850
I20240503 02:13:14.674964 15811 raft_consensus.cc:2781] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:14.675117 15811 raft_consensus.cc:482] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:14.672637 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:14.675237 15811 raft_consensus.cc:3037] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:13:14.677301 12840 leader_election.cc:341] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.677642 15703 leader_election.cc:288] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:14.679778 15811 raft_consensus.cc:504] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.683323 15811 leader_election.cc:288] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:14.683679 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
W20240503 02:13:14.684352 13107 leader_election.cc:341] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.685343 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "818345467bd9400aa5fdd875b59f3249" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:14.685535 10490 raft_consensus.cc:2362] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1. Re-sending same reply.
I20240503 02:13:14.685695 15973 tablet_bootstrap.cc:492] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:14.686614 15715 raft_consensus.cc:504] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.687572 15643 raft_consensus.cc:686] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:14.687938 15973 tablet_bootstrap.cc:654] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:14.687983 15643 consensus_queue.cc:237] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.688789 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:14.689055 12908 raft_consensus.cc:2379] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f93b103af083471b8e8c0376e9148aad in current term 1: Already voted for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in this term.
I20240503 02:13:14.690979 13107 leader_election.cc:302] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:14.691483 15811 raft_consensus.cc:2726] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:13:14.694062 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:14.694864 15715 leader_election.cc:288] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
W20240503 02:13:14.695829 12840 leader_election.cc:341] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:14.697599 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:14.697892 13172 raft_consensus.cc:2379] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in current term 1: Already voted for candidate f93b103af083471b8e8c0376e9148aad in this term.
I20240503 02:13:14.698297 12843 leader_election.cc:302] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:14.698820 15709 raft_consensus.cc:2726] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20240503 02:13:14.694494 10357 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.661804 9947 catalog_manager.cc:5484] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.703019 9951 catalog_manager.cc:5484] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:14.703595 15973 tablet_bootstrap.cc:492] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:14.703845 15973 ts_tablet_manager.cc:1367] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240503 02:13:14.704895 15973 raft_consensus.cc:348] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.705303 15973 raft_consensus.cc:374] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.705462 15973 raft_consensus.cc:729] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.705838 15973 consensus_queue.cc:260] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:14.706547 15973 ts_tablet_manager.cc:1398] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:14.716729 10090 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.726256 15629 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.748157 15629 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:14.753487 10092 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:14.764220 15874 raft_consensus.cc:1050] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:14.765565 15874 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:14.768260 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30] alter complete (version 1)
I20240503 02:13:14.773036 12906 consensus_queue.cc:237] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:14.776713 10491 raft_consensus.cc:1261] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:14.777844 15715 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:14.778996 12908 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:14.781805 15711 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.782382 10491 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.784020 10357 raft_consensus.cc:1261] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:14.785274 15990 consensus_queue.cc:1035] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:14.786701 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f1f1661a75a74746a4485b88b8e5b185 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:14.787698 10357 raft_consensus.cc:2932] T f1f1661a75a74746a4485b88b8e5b185 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.793466 9952 catalog_manager.cc:5484] T f1f1661a75a74746a4485b88b8e5b185 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.818262 15978 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:14.819062 15978 tablet.cc:1620] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:14.819408 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet f1f1661a75a74746a4485b88b8e5b185 with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:14.821033 15943 raft_consensus.cc:1050] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:14.822510 15943 consensus_queue.cc:237] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:14.836277 9951 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet e1784f1f151e4617aaff8ed526ecd316 (table rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30]). Expected version 1 got 0
I20240503 02:13:14.841840 15956 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:14.842994 15956 ts_tablet_manager.cc:1367] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.238s user 0.034s sys 0.040s
I20240503 02:13:14.843999 15956 raft_consensus.cc:348] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.844400 15956 raft_consensus.cc:374] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:14.844550 15956 raft_consensus.cc:729] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:14.844895 15956 consensus_queue.cc:260] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:14.845062 10227 raft_consensus.cc:1261] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.845171 13172 raft_consensus.cc:1261] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.846011 15943 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.846381 15613 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.850718 15956 ts_tablet_manager.cc:1398] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20240503 02:13:14.857147 10091 raft_consensus.cc:1261] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:14.859189 15943 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:14.836740 9943 catalog_manager.cc:5484] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:14.872366 10357 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:14.873890 15952 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:14.875473 12908 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:14.876209 15952 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:14.883586 9946 catalog_manager.cc:5484] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:14.885641 15994 log.cc:924] Time spent T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: Append to log took a long time: real 0.153s user 0.000s sys 0.000s
I20240503 02:13:14.883914 15613 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.892529 16004 tablet_replica.cc:330] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:14.892802 16004 raft_consensus.cc:2229] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:14.893056 16004 raft_consensus.cc:2258] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:14.894170 16004 ts_tablet_manager.cc:1875] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:14.900852 10359 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:14.911335 10224 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.915082 10092 consensus_queue.cc:237] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:14.918392 10091 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.920130 13168 raft_consensus.cc:1261] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:14.920166 10359 raft_consensus.cc:1261] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:14.920887 15703 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:14.921667 15703 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:14.925204 15991 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.929428 10359 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.929898 13168 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.937809 9950 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:14.940544 9947 catalog_manager.cc:5484] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.941802 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:14.943066 13172 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:14.946803 16004 ts_tablet_manager.cc:1888] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:14.946954 16004 log.cc:1196] T f1f1661a75a74746a4485b88b8e5b185 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/f1f1661a75a74746a4485b88b8e5b185
I20240503 02:13:14.951551 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:14.951301 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet f1f1661a75a74746a4485b88b8e5b185 (table rebalancer_extra_table_10 [id=3b08f4c08894454fbcdee44b3b6e712d]) successfully deleted
I20240503 02:13:14.954519 9946 catalog_manager.cc:5576] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119) does not have the latest schema for tablet e1784f1f151e4617aaff8ed526ecd316 (table rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30]). Expected version 2 got 1
I20240503 02:13:14.957954 15952 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.963764 9951 catalog_manager.cc:5484] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:14.975917 10357 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.984269 10359 consensus_queue.cc:237] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:14.984608 12906 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:14.987782 15978 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:14.988417 15978 tablet.cc:1620] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: MemRowSet was empty: no flush needed.
I20240503 02:13:15.004684 10089 raft_consensus.cc:1261] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:15.004530 10227 consensus_queue.cc:237] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:15.005864 15613 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:15.006655 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=7ebb2429115e48e59c96b61349c58e30] alter complete (version 2)
I20240503 02:13:14.994391 9951 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:15.010970 16014 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:15.015350 12906 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:15.017021 16003 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:15.021836 10092 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:15.023653 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ee635c5c4d2c4befa0283cb8226f28d7 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:15.024986 16003 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:15.027212 15740 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:15.030851 16016 tablet_replica.cc:330] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:15.029989 10089 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:15.031556 16016 raft_consensus.cc:2229] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:15.033984 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 13e83ce8820b491992c041a8c21fa3df with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:15.034440 16016 raft_consensus.cc:2258] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.036607 16020 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:15.036787 12907 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:15.040951 16020 tablet.cc:1620] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: MemRowSet was empty: no flush needed.
I20240503 02:13:15.051524 9950 catalog_manager.cc:5484] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:15.056560 13166 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:15.057793 15980 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:15.058460 15980 tablet.cc:1620] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:15.061026 13168 raft_consensus.cc:1261] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:15.066026 15943 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:15.066583 16010 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:15.070631 13166 raft_consensus.cc:2932] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:15.074174 9951 catalog_manager.cc:5484] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:15.076027 13040 raft_consensus.cc:3037] T 8710d52bcd6b45bdb4babf0ca987486c P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.076661 15703 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.079957 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e83ce8820b491992c041a8c21fa3df with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:15.082250 15989 consensus_queue.cc:1035] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.089226 15878 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20240503 02:13:15.092264 10161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.65:38119, user_credentials={real_user=slave}} blocked reactor thread for 49224.8us
I20240503 02:13:15.094820 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet ee635c5c4d2c4befa0283cb8226f28d7 with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:15.095185 15941 raft_consensus.cc:482] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:15.095350 15941 raft_consensus.cc:504] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:15.096323 15941 leader_election.cc:288] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:15.097716 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:15.098169 13172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:15.098534 12908 raft_consensus.cc:2379] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6ec2f1283994a3089741715f811c850 in current term 1: Already voted for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in this term.
I20240503 02:13:15.098603 13172 raft_consensus.cc:2379] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c6ec2f1283994a3089741715f811c850 in current term 1: Already voted for candidate f93b103af083471b8e8c0376e9148aad in this term.
I20240503 02:13:15.099617 10424 leader_election.cc:302] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ec2f1283994a3089741715f811c850; no voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:15.104720 15941 raft_consensus.cc:3037] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.109437 15941 raft_consensus.cc:2726] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:15.095047 16016 ts_tablet_manager.cc:1875] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:15.109568 13168 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:15.122931 10089 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:15.124346 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:15.139070 15980 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:15.139053 15978 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:15.145056 16020 tablet.cc:1722] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:15.163131 16016 ts_tablet_manager.cc:1888] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:15.163564 16016 log.cc:1196] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/2fadc7e9c7604b5d8f7b8751b6ca78a7
I20240503 02:13:15.164585 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 (table rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]) successfully deleted
I20240503 02:13:15.167292 16025 tablet_replica.cc:330] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:15.167686 16025 raft_consensus.cc:2229] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:15.168273 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=7ebb2429115e48e59c96b61349c58e30] alter complete (version 3)
I20240503 02:13:15.168190 16025 raft_consensus.cc:2258] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.176623 16025 ts_tablet_manager.cc:1875] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:15.214622 16016 tablet_replica.cc:330] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:15.215777 16016 raft_consensus.cc:2229] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:15.222499 15437 consensus_queue.cc:1035] T 8710d52bcd6b45bdb4babf0ca987486c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.269204 16016 raft_consensus.cc:2258] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.271517 16016 ts_tablet_manager.cc:1875] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:15.276563 9952 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:15.276894 9952 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:15.281042 9952 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet e1784f1f151e4617aaff8ed526ecd316
I20240503 02:13:15.282181 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet e1784f1f151e4617aaff8ed526ecd316 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:15 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:15.283954 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet e1784f1f151e4617aaff8ed526ecd316 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:15 UTC) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:15.284361 16004 tablet_replica.cc:330] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:15.288033 16004 raft_consensus.cc:2229] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:15.288317 16004 raft_consensus.cc:2258] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.289557 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet e1784f1f151e4617aaff8ed526ecd316 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:15 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:15.289682 16004 ts_tablet_manager.cc:1875] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:15.303849 16004 ts_tablet_manager.cc:1888] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:15.304019 16004 log.cc:1196] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/e1784f1f151e4617aaff8ed526ecd316
I20240503 02:13:15.310874 16025 ts_tablet_manager.cc:1888] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:15.312803 16025 log.cc:1196] T 13e83ce8820b491992c041a8c21fa3df P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/13e83ce8820b491992c041a8c21fa3df
I20240503 02:13:15.315204 16004 ts_tablet_manager.cc:1909] T e1784f1f151e4617aaff8ed526ecd316 P 031a0afb01b6469299fa829fa3467ef7: Deleting consensus metadata
I20240503 02:13:15.318209 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 13e83ce8820b491992c041a8c21fa3df (table rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]) successfully deleted
I20240503 02:13:15.313697 16025 tablet_replica.cc:330] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:15.320924 16025 raft_consensus.cc:2229] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:15.320947 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet e1784f1f151e4617aaff8ed526ecd316 (table rebalancer_dynamic_table_new_name [id=7ebb2429115e48e59c96b61349c58e30]) successfully deleted
I20240503 02:13:15.322870 16025 raft_consensus.cc:2258] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.324275 16043 tablet_replica.cc:330] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:15.324715 16043 raft_consensus.cc:2229] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:15.325315 16043 raft_consensus.cc:2258] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:15.326479 16043 ts_tablet_manager.cc:1875] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:15.328298 16025 ts_tablet_manager.cc:1875] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:15.333876 16016 ts_tablet_manager.cc:1888] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:15.334113 16016 log.cc:1196] T ee635c5c4d2c4befa0283cb8226f28d7 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/ee635c5c4d2c4befa0283cb8226f28d7
I20240503 02:13:15.335347 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet ee635c5c4d2c4befa0283cb8226f28d7 (table rebalance_test_table_15 [id=9edc40d1530544bda86cbe026c22a8d7]) successfully deleted
I20240503 02:13:15.342258 16043 ts_tablet_manager.cc:1888] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:15.343559 16043 log.cc:1196] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/e1784f1f151e4617aaff8ed526ecd316
I20240503 02:13:15.344266 16043 ts_tablet_manager.cc:1909] T e1784f1f151e4617aaff8ed526ecd316 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:13:15.345978 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet e1784f1f151e4617aaff8ed526ecd316 (table rebalancer_dynamic_table_new_name [id=7ebb2429115e48e59c96b61349c58e30]) successfully deleted
I20240503 02:13:15.360059 16025 ts_tablet_manager.cc:1888] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:15.360404 16025 log.cc:1196] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/e1784f1f151e4617aaff8ed526ecd316
I20240503 02:13:15.361006 16025 ts_tablet_manager.cc:1909] T e1784f1f151e4617aaff8ed526ecd316 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:13:15.362591 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet e1784f1f151e4617aaff8ed526ecd316 (table rebalancer_dynamic_table_new_name [id=7ebb2429115e48e59c96b61349c58e30]) successfully deleted
I20240503 02:13:15.385047 9951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:15.398270 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e]), partition=
I20240503 02:13:15.398726 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e]), partition=
I20240503 02:13:15.398782 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.399158 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.401499 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e]), partition=
I20240503 02:13:15.401937 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:15.414070 16055 tablet_bootstrap.cc:492] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:15.416570 16055 tablet_bootstrap.cc:654] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.421317 16056 tablet_bootstrap.cc:492] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:15.423918 16056 tablet_bootstrap.cc:654] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.432576 16055 tablet_bootstrap.cc:492] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:15.433128 16060 tablet_bootstrap.cc:492] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:15.432948 16055 ts_tablet_manager.cc:1367] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.008s
I20240503 02:13:15.436717 16055 raft_consensus.cc:348] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.437223 16055 raft_consensus.cc:374] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.437325 16055 raft_consensus.cc:729] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.437590 16055 consensus_queue.cc:260] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.439425 16060 tablet_bootstrap.cc:654] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:15.443599 16055 ts_tablet_manager.cc:1398] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20240503 02:13:15.446286 16056 tablet_bootstrap.cc:492] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:15.446602 16056 ts_tablet_manager.cc:1367] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.000s
I20240503 02:13:15.447767 16060 tablet_bootstrap.cc:492] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:15.447593 16056 raft_consensus.cc:348] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.447867 16056 raft_consensus.cc:374] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.448030 16056 raft_consensus.cc:729] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.448063 16060 ts_tablet_manager.cc:1367] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.004s
I20240503 02:13:15.448405 16056 consensus_queue.cc:260] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.449404 16056 ts_tablet_manager.cc:1398] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:15.451051 16060 raft_consensus.cc:348] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.451437 16060 raft_consensus.cc:374] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:15.451568 16060 raft_consensus.cc:729] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:15.452004 16060 consensus_queue.cc:260] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.454777 16060 ts_tablet_manager.cc:1398] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.007s user 0.002s sys 0.001s
I20240503 02:13:15.456784 15943 raft_consensus.cc:482] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:15.456986 15943 raft_consensus.cc:504] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.458051 15943 leader_election.cc:288] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:15.460179 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575fc57cd1624a0f8aaabfc5cf91e5f9" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:15.460659 12908 raft_consensus.cc:2454] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:15.462440 10292 leader_election.cc:302] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:15.463922 15744 raft_consensus.cc:2781] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:15.464035 15744 raft_consensus.cc:482] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:15.464104 15744 raft_consensus.cc:3037] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.467446 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575fc57cd1624a0f8aaabfc5cf91e5f9" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:15.467825 10227 raft_consensus.cc:2454] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:15.469310 15744 raft_consensus.cc:504] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.484122 15744 leader_election.cc:288] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:15.485949 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575fc57cd1624a0f8aaabfc5cf91e5f9" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:15.486281 12908 raft_consensus.cc:3037] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.489189 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "575fc57cd1624a0f8aaabfc5cf91e5f9" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:15.489485 10227 raft_consensus.cc:3037] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:15.492069 10224 consensus_queue.cc:237] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:15.507830 12908 raft_consensus.cc:2454] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:15.508605 10227 raft_consensus.cc:2454] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:15.508661 10292 leader_election.cc:302] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:15.509022 15744 raft_consensus.cc:2781] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:15.509213 15744 raft_consensus.cc:686] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:13:15.509985 15744 consensus_queue.cc:237] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:15.510916 10357 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:15.512604 10491 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:15.513723 10162 consensus_peers.cc:485] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ceb177d6952e4c978c8be93b5b026952. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:15.515383 16003 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:15.516052 16003 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:15.521540 16003 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:15.529517 9951 catalog_manager.cc:5484] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:15.534288 9946 catalog_manager.cc:5484] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:15.538271 10359 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:15.538806 10491 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:15.602517 13039 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "db2da39193164d1ebf830282258d47d7"
dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:47510
I20240503 02:13:15.602826 13039 raft_consensus.cc:595] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:15.655709 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:15.663587 12908 raft_consensus.cc:1261] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:15.664251 10224 raft_consensus.cc:1261] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:15.665689 15613 consensus_queue.cc:1035] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.668511 15613 consensus_queue.cc:1035] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:15.681802 9952 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (table rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e]). Expected version 1 got 0
I20240503 02:13:15.688163 16115 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:15.688843 16115 tablet.cc:1620] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: MemRowSet was empty: no flush needed.
I20240503 02:13:15.690356 9952 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (table rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e]). Expected version 1 got 0
I20240503 02:13:15.687377 16116 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:15.692067 16116 tablet.cc:1620] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:15.695658 9946 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (table rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e]). Expected version 1 got 0
I20240503 02:13:15.700320 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e] alter complete (version 1)
I20240503 02:13:15.710134 16119 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:15.711174 16119 tablet.cc:1620] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:15.720590 9952 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (table rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e]). Expected version 1 got 0
I20240503 02:13:15.769013 13039 consensus_queue.cc:237] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:15.774466 13166 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:15.776038 15952 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:15.776710 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 317ce62718cb412bbf04569cab14309f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:15.777643 10089 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:15.779336 15952 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:15.782610 15952 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:15.783068 13166 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:15.790027 9946 catalog_manager.cc:5484] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:15.809664 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:15.812259 10089 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:15.826031 16115 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:15.826692 16115 tablet.cc:1620] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: MemRowSet was empty: no flush needed.
I20240503 02:13:15.826951 16116 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:15.827514 16116 tablet.cc:1620] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:15.827750 16119 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:15.828312 16119 tablet.cc:1620] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:15.846146 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=904072b7f094477cb29ffc20774b161e] alter complete (version 2)
I20240503 02:13:15.864264 10227 consensus_queue.cc:237] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:15.866888 10359 raft_consensus.cc:1261] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:15.867614 16029 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:15.867774 10491 raft_consensus.cc:1261] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:15.868731 15740 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:15.869081 10159 consensus_peers.cc:485] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2fb90eef6224ed1a1dcc57c71683bc3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:15.873910 16029 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:15.878640 10359 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:15.880241 10491 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:15.881006 9951 catalog_manager.cc:5484] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:15.881700 15874 raft_consensus.cc:979] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: : Instructing follower 47f901e617024baaa1865d12261ef84f to start an election
I20240503 02:13:15.881897 16010 raft_consensus.cc:1067] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Signalling peer 47f901e617024baaa1865d12261ef84f to start an election
I20240503 02:13:15.882730 10357 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "db2da39193164d1ebf830282258d47d7"
dest_uuid: "47f901e617024baaa1865d12261ef84f"
from {username='slave'} at 127.4.5.70:45605
I20240503 02:13:15.883009 10357 raft_consensus.cc:482] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:15.883122 10357 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:15.887612 10357 raft_consensus.cc:504] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:15.889521 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:15.889793 13039 raft_consensus.cc:3032] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:15.889889 13039 raft_consensus.cc:729] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:15.890223 13039 consensus_queue.cc:260] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:15.890686 13039 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:15.905805 10357 leader_election.cc:288] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 2 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:15.900368 13039 raft_consensus.cc:2454] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 2.
I20240503 02:13:15.906284 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:15.906569 13168 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:15.909691 13168 raft_consensus.cc:2454] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 2.
I20240503 02:13:15.910162 10295 leader_election.cc:302] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, 8d454a3607d042dc9559ed59c3ccd84b, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:15.910539 16014 raft_consensus.cc:2781] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:15.910741 16014 raft_consensus.cc:686] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 2 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:13:15.911203 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:15.911144 16014 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:15.911492 12908 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:15.916317 12908 raft_consensus.cc:2454] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 2.
I20240503 02:13:15.916895 9952 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 1 to 2, leader changed from 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 2 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:15.937453 10227 consensus_queue.cc:237] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:15.942977 10357 raft_consensus.cc:1261] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:15.943549 16029 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:15.944528 10162 consensus_peers.cc:485] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be61de73e08c4998a51f02cafa46d421. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:15.948339 10491 raft_consensus.cc:1261] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:15.949025 16029 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:15.952286 9952 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:15.956552 15740 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:15.963223 10357 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:15.968041 16115 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:15.969152 16119 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:15.976720 9952 catalog_manager.cc:5484] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:15.983610 10491 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:15.984773 9951 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=904072b7f094477cb29ffc20774b161e] alter complete (version 3)
I20240503 02:13:15.995276 16116 tablet.cc:1722] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:16.072381 16164 ts_tablet_manager.cc:902] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:16.082943 16164 tablet_copy_client.cc:308] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:16.083837 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet ceb177d6952e4c978c8be93b5b026952 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:44307)
I20240503 02:13:16.084043 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet ceb177d6952e4c978c8be93b5b026952 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:44307: session id = f93b103af083471b8e8c0376e9148aad-ceb177d6952e4c978c8be93b5b026952
I20240503 02:13:16.087023 10248 tablet_copy_source_session.cc:205] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:16.088760 16164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb177d6952e4c978c8be93b5b026952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.088733 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:16.089002 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:16.093253 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 575fc57cd1624a0f8aaabfc5cf91e5f9
I20240503 02:13:16.094506 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:16 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:16.095417 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:16 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:16.095736 16175 tablet_replica.cc:330] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:16.096014 16175 raft_consensus.cc:2229] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.096350 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:16 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:16.096814 16176 tablet_replica.cc:330] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:16.096961 16164 tablet_copy_client.cc:783] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:16.097321 16164 tablet_copy_client.cc:647] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:16.097509 16175 raft_consensus.cc:2258] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.097177 16176 raft_consensus.cc:2229] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:16.097880 16177 tablet_replica.cc:330] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:16.098173 16177 raft_consensus.cc:2229] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:16.098482 16177 raft_consensus.cc:2258] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.099378 16176 raft_consensus.cc:2258] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:16.099584 16177 ts_tablet_manager.cc:1875] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:16.099612 16164 tablet_copy_client.cc:523] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:16.099762 16175 ts_tablet_manager.cc:1875] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:16.100548 16176 ts_tablet_manager.cc:1875] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:16.104029 16164 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:16.107473 16175 ts_tablet_manager.cc:1888] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:16.107712 16175 log.cc:1196] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/575fc57cd1624a0f8aaabfc5cf91e5f9
I20240503 02:13:16.108264 16175 ts_tablet_manager.cc:1909] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:13:16.107172 16177 ts_tablet_manager.cc:1888] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:16.108634 16177 log.cc:1196] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/575fc57cd1624a0f8aaabfc5cf91e5f9
I20240503 02:13:16.109598 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (table rebalancer_dynamic_table_new_name [id=904072b7f094477cb29ffc20774b161e]) successfully deleted
I20240503 02:13:16.109719 16177 ts_tablet_manager.cc:1909] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:13:16.110977 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (table rebalancer_dynamic_table_new_name [id=904072b7f094477cb29ffc20774b161e]) successfully deleted
I20240503 02:13:16.112114 16176 ts_tablet_manager.cc:1888] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:16.112254 16176 log.cc:1196] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/575fc57cd1624a0f8aaabfc5cf91e5f9
I20240503 02:13:16.112681 16176 ts_tablet_manager.cc:1909] T 575fc57cd1624a0f8aaabfc5cf91e5f9 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:13:16.113699 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 575fc57cd1624a0f8aaabfc5cf91e5f9 (table rebalancer_dynamic_table_new_name [id=904072b7f094477cb29ffc20774b161e]) successfully deleted
I20240503 02:13:16.142894 16164 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:16.143397 16164 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:16.143693 16164 ts_tablet_manager.cc:1367] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.040s user 0.028s sys 0.008s
I20240503 02:13:16.144599 16164 raft_consensus.cc:348] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:16.144905 16164 raft_consensus.cc:729] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:16.145172 16164 consensus_queue.cc:260] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:16.145807 16164 ts_tablet_manager.cc:1398] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:16.146773 10248 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-ceb177d6952e4c978c8be93b5b026952 received from {username='slave'} at 127.4.5.71:44307
I20240503 02:13:16.146924 10248 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session f93b103af083471b8e8c0376e9148aad-ceb177d6952e4c978c8be93b5b026952 on tablet ceb177d6952e4c978c8be93b5b026952 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:16.197003 9951 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:16.206018 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 4f26eba3d378482898feae92fe450062 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c47e8491710f4be28a7f2fc9f18927ef]), partition=
I20240503 02:13:16.206187 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 4f26eba3d378482898feae92fe450062 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c47e8491710f4be28a7f2fc9f18927ef]), partition=
I20240503 02:13:16.206512 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f26eba3d378482898feae92fe450062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.206655 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f26eba3d378482898feae92fe450062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.209698 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 4f26eba3d378482898feae92fe450062 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=c47e8491710f4be28a7f2fc9f18927ef]), partition=
I20240503 02:13:16.210134 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4f26eba3d378482898feae92fe450062. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.221925 16179 tablet_bootstrap.cc:492] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:16.223748 16179 tablet_bootstrap.cc:654] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.228722 16181 tablet_bootstrap.cc:492] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:16.229380 16179 tablet_bootstrap.cc:492] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:16.229658 16179 ts_tablet_manager.cc:1367] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.001s
I20240503 02:13:16.230597 16179 raft_consensus.cc:348] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.230962 16179 raft_consensus.cc:374] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.231112 16179 raft_consensus.cc:729] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.231182 16181 tablet_bootstrap.cc:654] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.231534 16179 consensus_queue.cc:260] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.237972 16179 ts_tablet_manager.cc:1398] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:13:16.245368 16182 tablet_bootstrap.cc:492] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:16.245965 16181 tablet_bootstrap.cc:492] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:16.246130 16181 ts_tablet_manager.cc:1367] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.005s
I20240503 02:13:16.247185 16181 raft_consensus.cc:348] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.247447 16181 raft_consensus.cc:374] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.247534 16181 raft_consensus.cc:729] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.247778 16181 consensus_queue.cc:260] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.249869 16181 ts_tablet_manager.cc:1398] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20240503 02:13:16.261567 16182 tablet_bootstrap.cc:654] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:16.262681 15952 raft_consensus.cc:482] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:16.262910 15952 raft_consensus.cc:504] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.263633 15952 leader_election.cc:288] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:16.264081 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f26eba3d378482898feae92fe450062" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:16.264358 10491 raft_consensus.cc:2454] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 0.
I20240503 02:13:16.264734 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f26eba3d378482898feae92fe450062" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:16.264839 12973 leader_election.cc:302] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters:
W20240503 02:13:16.266252 12975 leader_election.cc:341] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:16.266775 15874 raft_consensus.cc:2781] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:16.266917 15874 raft_consensus.cc:482] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:16.267014 15874 raft_consensus.cc:3037] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.270475 15874 raft_consensus.cc:504] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.272487 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f26eba3d378482898feae92fe450062" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:16.273027 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4f26eba3d378482898feae92fe450062" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:16.273317 10491 raft_consensus.cc:3037] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
W20240503 02:13:16.273456 12975 leader_election.cc:341] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:16.273839 15874 leader_election.cc:288] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:16.274343 16182 tablet_bootstrap.cc:492] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:16.274612 16182 ts_tablet_manager.cc:1367] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.006s
I20240503 02:13:16.275743 16182 raft_consensus.cc:348] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.276114 16182 raft_consensus.cc:374] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:16.276262 10491 raft_consensus.cc:2454] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:16.276290 16182 raft_consensus.cc:729] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:16.276832 16182 consensus_queue.cc:260] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.277016 12973 leader_election.cc:302] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850; no voters: eff8e8d11a4447218741bb994aa41082
I20240503 02:13:16.277818 16182 ts_tablet_manager.cc:1398] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:16.278988 15059 raft_consensus.cc:482] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:16.279181 15059 raft_consensus.cc:504] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:16.279958 15059 leader_election.cc:288] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:16.280885 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:16.281229 10491 raft_consensus.cc:2454] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:16.281450 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:16.281733 12908 raft_consensus.cc:2454] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:16.281769 13107 leader_election.cc:302] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:16.282213 15059 raft_consensus.cc:2781] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:16.282359 15059 raft_consensus.cc:482] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:16.282455 15059 raft_consensus.cc:3037] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:16.284091 15952 raft_consensus.cc:2781] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:16.284292 15952 raft_consensus.cc:686] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:16.284605 15952 consensus_queue.cc:237] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:16.288625 15059 raft_consensus.cc:504] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:16.292551 15059 leader_election.cc:288] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:16.293320 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:16.293596 12908 raft_consensus.cc:3037] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:16.293452 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "004a7ce056044d25ab68a44e51688ca3" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:16.293988 10491 raft_consensus.cc:3037] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:16.299279 12908 raft_consensus.cc:2454] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 2.
I20240503 02:13:16.299865 13107 leader_election.cc:302] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:16.300534 13039 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:16.303036 16014 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:16.310400 10491 raft_consensus.cc:2454] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 2.
I20240503 02:13:16.323438 15059 raft_consensus.cc:2781] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:16.324405 15059 raft_consensus.cc:686] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 2 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:13:16.325615 15059 consensus_queue.cc:237] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:16.358273 9951 catalog_manager.cc:5484] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 0 to 1, leader changed from <none> to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:16.362571 16184 ts_tablet_manager.cc:902] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:16.363116 16194 ts_tablet_manager.cc:902] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:16.366690 9952 catalog_manager.cc:5484] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 0 to 2, leader changed from <none> to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 2 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:16.368942 16194 tablet_copy_client.cc:308] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:16.369958 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet 317ce62718cb412bbf04569cab14309f from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:42943)
I20240503 02:13:16.370254 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet 317ce62718cb412bbf04569cab14309f from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:42943: session id = 47f901e617024baaa1865d12261ef84f-317ce62718cb412bbf04569cab14309f
I20240503 02:13:16.380579 13061 tablet_copy_source_session.cc:205] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:16.382495 16194 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 317ce62718cb412bbf04569cab14309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.398710 16184 tablet_copy_client.cc:308] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:16.399786 10248 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet d2fb90eef6224ed1a1dcc57c71683bc3 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:52743)
I20240503 02:13:16.399991 10248 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet d2fb90eef6224ed1a1dcc57c71683bc3 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:52743: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-d2fb90eef6224ed1a1dcc57c71683bc3
I20240503 02:13:16.402838 10248 tablet_copy_source_session.cc:205] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:16.406353 16194 tablet_copy_client.cc:783] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:16.406695 16194 tablet_copy_client.cc:647] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:16.408813 16194 tablet_copy_client.cc:523] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:16.412225 16184 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2fb90eef6224ed1a1dcc57c71683bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.415221 16194 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:16.443495 13168 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:16.445920 15744 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:16.460294 16216 ts_tablet_manager.cc:902] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:16.462039 16184 tablet_copy_client.cc:783] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:16.462529 16184 tablet_copy_client.cc:647] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:16.462857 16216 tablet_copy_client.cc:308] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:16.464226 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet be61de73e08c4998a51f02cafa46d421 from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:38111)
I20240503 02:13:16.464538 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet be61de73e08c4998a51f02cafa46d421 from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:38111: session id = 8d454a3607d042dc9559ed59c3ccd84b-be61de73e08c4998a51f02cafa46d421
I20240503 02:13:16.466238 16194 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:16.466732 16194 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:16.467015 16194 ts_tablet_manager.cc:1367] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.008s
I20240503 02:13:16.467739 16194 raft_consensus.cc:348] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:16.467974 16194 raft_consensus.cc:729] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:16.468201 16194 consensus_queue.cc:260] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:16.468815 16194 ts_tablet_manager.cc:1398] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:16.471062 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-317ce62718cb412bbf04569cab14309f received from {username='slave'} at 127.4.5.67:42943
I20240503 02:13:16.471225 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session 47f901e617024baaa1865d12261ef84f-317ce62718cb412bbf04569cab14309f on tablet 317ce62718cb412bbf04569cab14309f with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:16.473317 10246 tablet_copy_source_session.cc:205] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:16.474361 16184 tablet_copy_client.cc:523] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:16.479889 16216 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet be61de73e08c4998a51f02cafa46d421. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:16.485002 12908 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:16.494022 9951 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:16.496599 16184 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:16.512813 16226 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:16.540750 13168 raft_consensus.cc:1203] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:16.548238 10227 raft_consensus.cc:3037] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:16.551468 10227 raft_consensus.cc:1261] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:16.552361 15874 consensus_queue.cc:1035] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.553318 16216 tablet_copy_client.cc:783] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:16.553668 16216 tablet_copy_client.cc:647] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:16.560168 16216 tablet_copy_client.cc:523] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:16.560736 10491 raft_consensus.cc:1261] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:16.562347 16010 consensus_queue.cc:1035] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.573321 16216 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:16.578902 9952 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet 4f26eba3d378482898feae92fe450062 (table rebalancer_dynamic_table [id=c47e8491710f4be28a7f2fc9f18927ef]). Expected version 1 got 0
I20240503 02:13:16.598191 16184 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:16.598747 16184 tablet_bootstrap.cc:492] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:16.599109 16184 ts_tablet_manager.cc:1367] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.103s user 0.049s sys 0.005s
I20240503 02:13:16.600111 16184 raft_consensus.cc:348] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:16.600394 16184 raft_consensus.cc:729] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:16.600646 16184 consensus_queue.cc:260] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:16.601202 16184 ts_tablet_manager.cc:1398] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:16.602972 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-d2fb90eef6224ed1a1dcc57c71683bc3 received from {username='slave'} at 127.4.5.72:52743
I20240503 02:13:16.603168 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-d2fb90eef6224ed1a1dcc57c71683bc3 on tablet d2fb90eef6224ed1a1dcc57c71683bc3 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:16.616362 16236 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:16.617370 16236 tablet.cc:1620] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:16.618213 16237 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:16.618870 16237 tablet.cc:1620] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:13:16.622717 9952 catalog_manager.cc:5576] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025) does not have the latest schema for tablet 4f26eba3d378482898feae92fe450062 (table rebalancer_dynamic_table [id=c47e8491710f4be28a7f2fc9f18927ef]). Expected version 1 got 0
I20240503 02:13:16.632090 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=c47e8491710f4be28a7f2fc9f18927ef] alter complete (version 1)
I20240503 02:13:16.638494 9952 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet 4f26eba3d378482898feae92fe450062 (table rebalancer_dynamic_table [id=c47e8491710f4be28a7f2fc9f18927ef]). Expected version 1 got 0
I20240503 02:13:16.639655 16243 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:16.640409 16243 tablet.cc:1620] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:16.694003 16216 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:16.694602 16216 tablet_bootstrap.cc:492] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:16.694904 16216 ts_tablet_manager.cc:1367] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.122s user 0.054s sys 0.014s
I20240503 02:13:16.695724 16216 raft_consensus.cc:348] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:16.696043 16216 raft_consensus.cc:729] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:16.696307 16216 consensus_queue.cc:260] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:16.697230 16216 ts_tablet_manager.cc:1398] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:16.711266 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-be61de73e08c4998a51f02cafa46d421 received from {username='slave'} at 127.4.5.70:38111
I20240503 02:13:16.711562 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-be61de73e08c4998a51f02cafa46d421 on tablet be61de73e08c4998a51f02cafa46d421 with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:16.746551 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:16.762462 16236 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:16.763099 16236 tablet.cc:1620] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:16.764432 16243 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:16.765082 16243 tablet.cc:1620] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:16.776526 9951 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=c47e8491710f4be28a7f2fc9f18927ef] alter complete (version 2)
I20240503 02:13:16.776513 16237 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:16.777103 16237 tablet.cc:1620] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: MemRowSet was empty: no flush needed.
I20240503 02:13:16.779191 12908 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 02:13:16.783859 15059 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.790036 10357 raft_consensus.cc:1203] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:16.805572 10489 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20240503 02:13:16.806442 15059 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:16.856758 13302 raft_consensus.cc:1203] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:16.879360 9952 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:16.891782 16236 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:16.892334 16243 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:16.895339 16237 tablet.cc:1722] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:16.904497 9950 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=c47e8491710f4be28a7f2fc9f18927ef] alter complete (version 3)
I20240503 02:13:16.975983 13040 raft_consensus.cc:1203] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:17.000078 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:17.010164 9946 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:17.010445 9946 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:17.018206 12887 tablet_service.cc:1467] Processing CreateTablet for tablet ccd3faa7afbb4b92a4a4c4645e477771 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.018375 10207 tablet_service.cc:1467] Processing CreateTablet for tablet ccd3faa7afbb4b92a4a4c4645e477771 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.018810 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd3faa7afbb4b92a4a4c4645e477771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.018893 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd3faa7afbb4b92a4a4c4645e477771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.020154 13153 tablet_service.cc:1467] Processing CreateTablet for tablet e8819ae761d24f9a9b6f5570a14cf3c3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.020840 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8819ae761d24f9a9b6f5570a14cf3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.024678 10074 tablet_service.cc:1467] Processing CreateTablet for tablet ccd3faa7afbb4b92a4a4c4645e477771 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.025210 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd3faa7afbb4b92a4a4c4645e477771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.027043 16276 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:17.029099 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 42c4f4a0f3d8448aa82becd57e5c49ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.030952 9946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 4f26eba3d378482898feae92fe450062
I20240503 02:13:17.032728 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f26eba3d378482898feae92fe450062 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:17 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:17.032889 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 42c4f4a0f3d8448aa82becd57e5c49ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.033124 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f26eba3d378482898feae92fe450062 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:17 UTC) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:17.033429 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c4f4a0f3d8448aa82becd57e5c49ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.033748 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c4f4a0f3d8448aa82becd57e5c49ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.034556 10073 tablet_service.cc:1467] Processing CreateTablet for tablet e8819ae761d24f9a9b6f5570a14cf3c3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.034556 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 42c4f4a0f3d8448aa82becd57e5c49ed (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.036470 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8819ae761d24f9a9b6f5570a14cf3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.036883 10338 tablet_service.cc:1467] Processing CreateTablet for tablet e8819ae761d24f9a9b6f5570a14cf3c3 (DEFAULT_TABLE table=rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:17.037082 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c4f4a0f3d8448aa82becd57e5c49ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.037420 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8819ae761d24f9a9b6f5570a14cf3c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.045181 10206 tablet_service.cc:1514] Processing DeleteTablet for tablet 4f26eba3d378482898feae92fe450062 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:17 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:17.064731 16284 tablet_bootstrap.cc:492] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:17.065227 16282 tablet_replica.cc:330] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:17.066056 16282 raft_consensus.cc:2229] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.067693 16282 raft_consensus.cc:2258] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.073851 16282 ts_tablet_manager.cc:1875] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:17.074734 16276 tablet_bootstrap.cc:654] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.075336 16278 tablet_replica.cc:330] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:17.075896 16278 raft_consensus.cc:2229] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:17.076486 16278 raft_consensus.cc:2258] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.077836 16278 ts_tablet_manager.cc:1875] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:17.078456 16279 tablet_replica.cc:330] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:17.078490 16281 tablet_bootstrap.cc:492] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:17.078774 16279 raft_consensus.cc:2229] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.079035 16279 raft_consensus.cc:2258] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.079648 16285 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:17.081634 16285 tablet_bootstrap.cc:654] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.083307 16279 ts_tablet_manager.cc:1875] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:17.085072 16284 tablet_bootstrap.cc:654] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.084435 16281 tablet_bootstrap.cc:654] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.088110 16283 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:17.089298 16276 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:17.091231 16283 tablet_bootstrap.cc:654] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.091500 16276 ts_tablet_manager.cc:1367] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.065s user 0.005s sys 0.000s
I20240503 02:13:17.094362 16282 ts_tablet_manager.cc:1888] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:17.095494 16284 tablet_bootstrap.cc:492] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:17.095741 16284 ts_tablet_manager.cc:1367] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20240503 02:13:17.096786 16284 raft_consensus.cc:348] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.097108 16284 raft_consensus.cc:374] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.097219 16284 raft_consensus.cc:729] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.097666 16284 consensus_queue.cc:260] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.098006 16276 raft_consensus.cc:348] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.098346 16276 raft_consensus.cc:374] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20240503 02:13:17.092936 10295 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 56181.1us
I20240503 02:13:17.101718 16278 ts_tablet_manager.cc:1888] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:17.104470 16276 raft_consensus.cc:729] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.105106 16278 log.cc:1196] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/4f26eba3d378482898feae92fe450062
I20240503 02:13:17.105850 16278 ts_tablet_manager.cc:1909] T 4f26eba3d378482898feae92fe450062 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:17.106199 16276 consensus_queue.cc:260] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.111433 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 4f26eba3d378482898feae92fe450062 (table rebalancer_dynamic_table_new_name [id=c47e8491710f4be28a7f2fc9f18927ef]) successfully deleted
I20240503 02:13:17.111594 16276 ts_tablet_manager.cc:1398] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:13:17.115278 16279 ts_tablet_manager.cc:1888] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:17.115615 16279 log.cc:1196] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/4f26eba3d378482898feae92fe450062
I20240503 02:13:17.115707 16284 ts_tablet_manager.cc:1398] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:13:17.116070 16283 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:17.116328 16283 ts_tablet_manager.cc:1367] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.000s
I20240503 02:13:17.116356 16281 tablet_bootstrap.cc:492] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:17.116365 16279 ts_tablet_manager.cc:1909] T 4f26eba3d378482898feae92fe450062 P c6ec2f1283994a3089741715f811c850: Deleting consensus metadata
I20240503 02:13:17.116544 16281 ts_tablet_manager.cc:1367] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.038s user 0.008s sys 0.000s
I20240503 02:13:17.117671 16284 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:17.117614 16281 raft_consensus.cc:348] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.117712 16283 raft_consensus.cc:348] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:17.117949 16283 raft_consensus.cc:374] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.118008 16281 raft_consensus.cc:374] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.118095 16283 raft_consensus.cc:729] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.118166 16281 raft_consensus.cc:729] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.118382 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 4f26eba3d378482898feae92fe450062 (table rebalancer_dynamic_table_new_name [id=c47e8491710f4be28a7f2fc9f18927ef]) successfully deleted
I20240503 02:13:17.118405 16283 consensus_queue.cc:260] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:17.118510 16281 consensus_queue.cc:260] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.119232 16283 ts_tablet_manager.cc:1398] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:17.119338 16284 tablet_bootstrap.cc:654] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.097801 16285 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:17.120079 16282 log.cc:1196] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/4f26eba3d378482898feae92fe450062
I20240503 02:13:17.120179 16285 ts_tablet_manager.cc:1367] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.003s
I20240503 02:13:17.120535 16282 ts_tablet_manager.cc:1909] T 4f26eba3d378482898feae92fe450062 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:13:17.121009 16285 raft_consensus.cc:348] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.121214 16285 raft_consensus.cc:374] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.121264 16285 raft_consensus.cc:729] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.121475 16285 consensus_queue.cc:260] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.121688 16283 tablet_bootstrap.cc:492] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:17.122007 16285 ts_tablet_manager.cc:1398] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:17.122040 16281 ts_tablet_manager.cc:1398] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:17.122903 16281 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:17.124373 16283 tablet_bootstrap.cc:654] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.127431 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 4f26eba3d378482898feae92fe450062 (table rebalancer_dynamic_table_new_name [id=c47e8491710f4be28a7f2fc9f18927ef]) successfully deleted
I20240503 02:13:17.132855 16281 tablet_bootstrap.cc:654] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.136132 16283 tablet_bootstrap.cc:492] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:17.136488 16283 ts_tablet_manager.cc:1367] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.000s
I20240503 02:13:17.137627 16283 raft_consensus.cc:348] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.137962 16283 raft_consensus.cc:374] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.138101 16283 raft_consensus.cc:729] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.138447 16283 consensus_queue.cc:260] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.139325 16283 ts_tablet_manager.cc:1398] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:17.140934 10359 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:17.143355 16284 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:17.143575 16284 ts_tablet_manager.cc:1367] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.005s
I20240503 02:13:17.144006 13168 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:17.144393 16284 raft_consensus.cc:348] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.144883 16226 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.144663 16284 raft_consensus.cc:374] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.145378 16284 raft_consensus.cc:729] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.145659 16284 consensus_queue.cc:260] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.145951 16281 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:17.145890 9952 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:17.146252 16281 ts_tablet_manager.cc:1367] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.000s
I20240503 02:13:17.147688 16284 ts_tablet_manager.cc:1398] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:17.147552 16281 raft_consensus.cc:348] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:17.147989 16281 raft_consensus.cc:374] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.148008 16284 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:17.148156 16281 raft_consensus.cc:729] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.148586 16281 consensus_queue.cc:260] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:17.149024 12908 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:17.151337 16281 ts_tablet_manager.cc:1398] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:17.152892 15740 raft_consensus.cc:1050] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:17.153786 15740 consensus_queue.cc:237] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:17.153980 16280 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.157832 13166 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.158180 10489 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.158560 16003 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.159183 16297 raft_consensus.cc:482] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.159343 16297 raft_consensus.cc:504] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.159773 13166 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.160753 16297 leader_election.cc:288] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:17.161289 16297 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.163798 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd3faa7afbb4b92a4a4c4645e477771" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:17.164036 10089 raft_consensus.cc:2454] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:17.165452 10161 leader_election.cc:302] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:17.165910 16003 raft_consensus.cc:2781] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.165999 16003 raft_consensus.cc:482] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.166067 16003 raft_consensus.cc:3037] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.165962 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd3faa7afbb4b92a4a4c4645e477771" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:17.166390 12908 raft_consensus.cc:2454] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:17.169387 16003 raft_consensus.cc:504] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.170125 16003 leader_election.cc:288] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:17.170547 12906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd3faa7afbb4b92a4a4c4645e477771" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:17.170851 12906 raft_consensus.cc:3037] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.170977 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ccd3faa7afbb4b92a4a4c4645e477771" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:17.171204 10089 raft_consensus.cc:3037] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.171626 16226 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.173956 12906 raft_consensus.cc:2454] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:17.174430 10089 raft_consensus.cc:2454] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:17.175082 10159 leader_election.cc:302] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:17.175405 16003 raft_consensus.cc:2781] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.175559 16003 raft_consensus.cc:686] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:17.175818 16003 consensus_queue.cc:237] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:17.178844 16284 tablet_bootstrap.cc:654] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.183104 12907 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.184600 10359 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.185045 9951 catalog_manager.cc:5484] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:17.185755 10207 tablet_service.cc:1467] Processing CreateTablet for tablet e8c934099ef34318bff62c90eec41c07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]), partition=
I20240503 02:13:17.186159 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c934099ef34318bff62c90eec41c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.186398 13020 tablet_service.cc:1467] Processing CreateTablet for tablet e8c934099ef34318bff62c90eec41c07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]), partition=
I20240503 02:13:17.186873 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c934099ef34318bff62c90eec41c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.188792 15740 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.189713 13172 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.192425 16210 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.196555 9952 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 5, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 2 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.200402 12887 tablet_service.cc:1467] Processing CreateTablet for tablet e8c934099ef34318bff62c90eec41c07 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]), partition=
I20240503 02:13:17.200932 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c934099ef34318bff62c90eec41c07. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:17.211756 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet db2da39193164d1ebf830282258d47d7 with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:17.213322 16278 tablet_replica.cc:330] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:17.213586 16278 raft_consensus.cc:2229] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.213794 16278 raft_consensus.cc:2258] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.215240 16278 ts_tablet_manager.cc:1875] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:17.217200 16276 tablet_bootstrap.cc:492] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:17.218924 16276 tablet_bootstrap.cc:654] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.222669 9950 catalog_manager.cc:5484] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.235600 16278 ts_tablet_manager.cc:1888] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 02:13:17.235924 16278 log.cc:1196] T db2da39193164d1ebf830282258d47d7 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/db2da39193164d1ebf830282258d47d7
I20240503 02:13:17.238818 10227 consensus_queue.cc:237] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:17.239948 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db2da39193164d1ebf830282258d47d7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:17.243882 16276 tablet_bootstrap.cc:492] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:17.244128 16284 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:17.244382 16276 ts_tablet_manager.cc:1367] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.010s
I20240503 02:13:17.244679 16284 ts_tablet_manager.cc:1367] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.097s user 0.007s sys 0.000s
I20240503 02:13:17.245714 16276 raft_consensus.cc:348] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.246130 15874 raft_consensus.cc:1050] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:17.246744 13172 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:17.246749 16284 raft_consensus.cc:348] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:17.246948 16284 raft_consensus.cc:374] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.246999 16284 raft_consensus.cc:729] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.247170 15874 consensus_queue.cc:237] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:17.247223 16284 consensus_queue.cc:260] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:17.247622 16276 raft_consensus.cc:374] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.247900 16276 raft_consensus.cc:729] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.248567 16210 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:17.251137 16284 ts_tablet_manager.cc:1398] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20240503 02:13:17.251276 16276 consensus_queue.cc:260] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.252285 16276 ts_tablet_manager.cc:1398] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20240503 02:13:17.261729 16210 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.265139 10357 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:17.266535 16010 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:17.267486 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ceb177d6952e4c978c8be93b5b026952 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:17.271672 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet db2da39193164d1ebf830282258d47d7 (table rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]) successfully deleted
I20240503 02:13:17.271953 16305 tablet_bootstrap.cc:492] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:17.274188 16305 tablet_bootstrap.cc:654] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.273736 13172 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.277026 13168 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:17.278039 15874 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:17.278730 10089 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:17.280294 15952 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:17.284427 9946 catalog_manager.cc:5484] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.290649 15952 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.294273 16305 tablet_bootstrap.cc:492] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:17.294466 16305 ts_tablet_manager.cc:1367] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.003s
I20240503 02:13:17.294994 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb177d6952e4c978c8be93b5b026952 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:17.295485 16305 raft_consensus.cc:348] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.295830 16305 raft_consensus.cc:374] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.295969 16305 raft_consensus.cc:729] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.296345 16305 consensus_queue.cc:260] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.297130 10091 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.305608 9946 catalog_manager.cc:5484] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:17.316495 16305 ts_tablet_manager.cc:1398] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240503 02:13:17.312178 13172 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.335444 16195 raft_consensus.cc:482] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.335717 16195 raft_consensus.cc:504] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.336418 16285 tablet_bootstrap.cc:492] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:17.337153 16195 leader_election.cc:288] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:17.338165 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8819ae761d24f9a9b6f5570a14cf3c3" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:13:17.338460 10354 raft_consensus.cc:2454] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:17.338517 16285 tablet_bootstrap.cc:654] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:17.338989 10028 leader_election.cc:302] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:13:17.339375 16195 raft_consensus.cc:2781] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.339493 16195 raft_consensus.cc:482] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.339576 16195 raft_consensus.cc:3037] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.348296 16195 raft_consensus.cc:504] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.338624 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8819ae761d24f9a9b6f5570a14cf3c3" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:17.350982 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8819ae761d24f9a9b6f5570a14cf3c3" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:17.351269 10354 raft_consensus.cc:3037] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.351294 13166 raft_consensus.cc:2454] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:17.353313 16195 leader_election.cc:288] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:17.353897 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8819ae761d24f9a9b6f5570a14cf3c3" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:17.354213 13168 raft_consensus.cc:3037] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.360954 13168 raft_consensus.cc:2454] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:17.361555 10029 leader_election.cc:302] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:17.361927 16195 raft_consensus.cc:2781] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.362052 16195 raft_consensus.cc:686] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:13:17.362349 16195 consensus_queue.cc:237] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } }
I20240503 02:13:17.363617 10354 raft_consensus.cc:2454] T e8819ae761d24f9a9b6f5570a14cf3c3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:17.377921 9951 catalog_manager.cc:5484] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:17.399224 15990 raft_consensus.cc:482] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.399538 15990 raft_consensus.cc:504] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.400666 15990 leader_election.cc:288] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:17.401803 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c934099ef34318bff62c90eec41c07" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:17.401985 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c934099ef34318bff62c90eec41c07" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:17.402334 13039 raft_consensus.cc:2454] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
W20240503 02:13:17.402732 12842 leader_election.cc:341] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:17.403625 12843 leader_election.cc:302] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: eff8e8d11a4447218741bb994aa41082
I20240503 02:13:17.404636 15990 raft_consensus.cc:2781] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.404759 15990 raft_consensus.cc:482] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.404875 15990 raft_consensus.cc:3037] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.407187 15740 raft_consensus.cc:1050] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:17.409646 15990 raft_consensus.cc:504] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.411360 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c934099ef34318bff62c90eec41c07" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:17.411962 15740 consensus_queue.cc:237] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
W20240503 02:13:17.412180 12842 leader_election.cc:341] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:17.413533 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c934099ef34318bff62c90eec41c07" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:17.413820 13039 raft_consensus.cc:3037] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.415653 13302 raft_consensus.cc:1261] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.416182 10354 raft_consensus.cc:1261] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.417053 16210 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.417701 16210 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.418638 13039 raft_consensus.cc:2454] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:17.419379 12843 leader_election.cc:302] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: eff8e8d11a4447218741bb994aa41082
I20240503 02:13:17.420390 16138 raft_consensus.cc:2781] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.420583 16138 raft_consensus.cc:686] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:17.420982 16138 consensus_queue.cc:237] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.428520 16003 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:17.432071 15990 leader_election.cc:288] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:17.433542 13302 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:17.443850 10359 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.455441 9950 catalog_manager.cc:5484] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:17.461272 15699 raft_consensus.cc:482] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.461526 15699 raft_consensus.cc:504] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:17.462430 15699 leader_election.cc:288] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:17.464864 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c4f4a0f3d8448aa82becd57e5c49ed" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:17.465256 13168 raft_consensus.cc:2454] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:17.466588 10029 leader_election.cc:302] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:17.467025 15699 raft_consensus.cc:2781] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:17.467128 15699 raft_consensus.cc:482] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:17.467234 15699 raft_consensus.cc:3037] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.471457 15699 raft_consensus.cc:504] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
I20240503 02:13:17.483457 15699 leader_election.cc:288] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:17.489069 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c4f4a0f3d8448aa82becd57e5c49ed" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:17.489362 13168 raft_consensus.cc:3037] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.492875 13168 raft_consensus.cc:2454] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:17.493584 10029 leader_election.cc:302] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:17.500322 16195 raft_consensus.cc:2781] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:17.500509 16195 raft_consensus.cc:686] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:13:17.500787 16195 consensus_queue.cc:237] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } }
W20240503 02:13:17.441673 10424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 236510us
I20240503 02:13:17.526782 13039 consensus_queue.cc:237] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:17.508224 16285 tablet_bootstrap.cc:492] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:17.538604 16285 ts_tablet_manager.cc:1367] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.202s user 0.001s sys 0.010s
I20240503 02:13:17.463675 10354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c4f4a0f3d8448aa82becd57e5c49ed" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:13:17.541134 10354 raft_consensus.cc:2454] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 0.
I20240503 02:13:17.480230 10355 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42c4f4a0f3d8448aa82becd57e5c49ed" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:17.542646 10355 raft_consensus.cc:3037] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.486742 10358 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:17.516045 10357 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.558256 10355 raft_consensus.cc:2454] T 42c4f4a0f3d8448aa82becd57e5c49ed P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
W20240503 02:13:17.564328 12840 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 140650us
I20240503 02:13:17.565063 10353 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:17.570173 16316 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:17.576568 10426 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.65:38119, user_credentials={real_user=slave}} blocked reactor thread for 376672us
I20240503 02:13:17.580895 9951 catalog_manager.cc:5484] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:17.582887 10489 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.584233 13172 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:17.585371 15874 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:17.580416 16297 raft_consensus.cc:1050] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:17.588407 16297 consensus_queue.cc:237] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:17.590299 16010 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.590608 10489 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:17.593052 13166 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.594767 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 317ce62718cb412bbf04569cab14309f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:17.594435 10489 raft_consensus.cc:1261] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.600273 16210 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:17.603899 9951 catalog_manager.cc:5484] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.611517 13039 raft_consensus.cc:1261] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.612814 15740 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.626377 10353 raft_consensus.cc:1261] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.627565 16297 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.642135 16285 raft_consensus.cc:348] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.642486 16285 raft_consensus.cc:374] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:17.642604 16285 raft_consensus.cc:729] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:17.642911 16285 consensus_queue.cc:260] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.643827 16285 ts_tablet_manager.cc:1398] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20240503 02:13:17.649652 10355 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:17.662379 15740 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:17.667822 10358 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
W20240503 02:13:17.668947 10159 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.68:35025, user_credentials={real_user=slave}} blocked reactor thread for 69241us
I20240503 02:13:17.672039 16210 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.674419 10489 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:17.686571 9946 catalog_manager.cc:5484] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:17.695652 9951 catalog_manager.cc:5484] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:17.701256 10074 tablet_service.cc:1514] Processing DeleteTablet for tablet 317ce62718cb412bbf04569cab14309f with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:17.716269 16315 tablet_replica.cc:330] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:17.716600 16315 raft_consensus.cc:2229] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.716871 16315 raft_consensus.cc:2258] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.720111 16315 ts_tablet_manager.cc:1875] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:17.724594 16343 tablet_replica.cc:330] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:17.725025 16343 raft_consensus.cc:2229] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:17.725390 16343 raft_consensus.cc:2258] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:17.726418 16343 ts_tablet_manager.cc:1875] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:17.730706 16297 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.740578 10489 raft_consensus.cc:1261] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:17.743531 16343 ts_tablet_manager.cc:1888] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:17.743877 16343 log.cc:1196] T 317ce62718cb412bbf04569cab14309f P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/317ce62718cb412bbf04569cab14309f
I20240503 02:13:17.749963 16195 consensus_queue.cc:1035] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.750926 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 317ce62718cb412bbf04569cab14309f (table rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]) successfully deleted
I20240503 02:13:17.761906 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:17.782677 9951 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]). Expected version 1 got 0
I20240503 02:13:17.798449 13039 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:17.808326 16297 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.808867 16003 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:17.825009 9950 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]). Expected version 1 got 0
I20240503 02:13:17.840649 10491 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:17.847178 16210 raft_consensus.cc:482] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:17.847416 16210 raft_consensus.cc:504] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:17.848340 16210 leader_election.cc:288] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:17.849627 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c934099ef34318bff62c90eec41c07" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:17.850029 13040 raft_consensus.cc:2379] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate eff8e8d11a4447218741bb994aa41082 in current term 1: Already voted for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in this term.
W20240503 02:13:17.851339 10424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 138037us
I20240503 02:13:17.858788 10227 raft_consensus.cc:3037] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:17.859283 16331 consensus_queue.cc:1035] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.862947 10227 raft_consensus.cc:1261] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:17.864210 13038 raft_consensus.cc:1261] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:17.865295 16138 consensus_queue.cc:1035] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.870584 16315 ts_tablet_manager.cc:1888] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:17.870561 16138 consensus_queue.cc:1035] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.870949 16315 log.cc:1196] T ceb177d6952e4c978c8be93b5b026952 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/ceb177d6952e4c978c8be93b5b026952
I20240503 02:13:17.874459 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet ceb177d6952e4c978c8be93b5b026952 (table rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]) successfully deleted
I20240503 02:13:17.880074 12908 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c934099ef34318bff62c90eec41c07" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:17.882536 10159 leader_election.cc:302] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: eff8e8d11a4447218741bb994aa41082; no voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:17.924661 10489 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:17.985718 16195 consensus_queue.cc:1035] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:17.997184 16366 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:17.998075 16365 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:18.002063 9950 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]). Expected version 1 got 0
I20240503 02:13:18.003541 16366 tablet.cc:1620] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:18.004484 16365 tablet.cc:1620] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:18.015878 9946 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]). Expected version 1 got 0
I20240503 02:13:18.029042 9951 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9] alter complete (version 1)
W20240503 02:13:18.042086 10427 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 134116us
I20240503 02:13:18.044864 16330 consensus_queue.cc:1035] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.086207 15740 raft_consensus.cc:2726] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:18.090092 9946 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]). Expected version 1 got 0
I20240503 02:13:18.117398 16373 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:18.118176 16373 tablet.cc:1620] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:18.122330 9950 catalog_manager.cc:5576] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397) does not have the latest schema for tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9]). Expected version 1 got 0
I20240503 02:13:18.169385 10224 consensus_queue.cc:237] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:18.172009 10491 raft_consensus.cc:1261] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:18.172569 13305 raft_consensus.cc:1261] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:18.175237 16297 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:18.175968 16210 consensus_queue.cc:1035] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:18.176710 9950 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:18.184811 16339 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:18.186141 10491 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:18.191318 13305 raft_consensus.cc:2932] T d2fb90eef6224ed1a1dcc57c71683bc3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:18.191859 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d2fb90eef6224ed1a1dcc57c71683bc3 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:18.195705 9951 catalog_manager.cc:5484] T d2fb90eef6224ed1a1dcc57c71683bc3 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:18.206830 16365 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:18.207481 16365 tablet.cc:1620] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:18.207409 16373 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:18.207911 16373 tablet.cc:1620] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: MemRowSet was empty: no flush needed.
I20240503 02:13:18.211402 16366 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:18.211915 16366 tablet.cc:1620] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:18.215147 10337 tablet_service.cc:1514] Processing DeleteTablet for tablet d2fb90eef6224ed1a1dcc57c71683bc3 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:18.215518 16315 tablet_replica.cc:330] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:18.215782 16315 raft_consensus.cc:2229] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:18.216140 16315 raft_consensus.cc:2258] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.217777 16315 ts_tablet_manager.cc:1875] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:18.224756 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=af0cf6e68d134dd6aae9305f33205bc9] alter complete (version 2)
I20240503 02:13:18.239342 16315 ts_tablet_manager.cc:1888] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:18.239703 16315 log.cc:1196] T d2fb90eef6224ed1a1dcc57c71683bc3 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/d2fb90eef6224ed1a1dcc57c71683bc3
I20240503 02:13:18.241025 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet d2fb90eef6224ed1a1dcc57c71683bc3 (table rebalance_test_table_3 [id=26020bab27604ae987d42513169cbf48]) successfully deleted
I20240503 02:13:18.253158 10227 consensus_queue.cc:237] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:18.255859 10358 raft_consensus.cc:1261] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:18.256341 10159 consensus_peers.cc:485] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0a5125fbc8ec4ba887abce94a84efbdc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.256477 10489 raft_consensus.cc:1261] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.256721 15740 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.257241 16210 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.259792 16339 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:18.260506 10358 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:18.266698 10489 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:18.267587 9950 catalog_manager.cc:5484] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 4, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:18.297878 10227 consensus_queue.cc:237] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:18.300395 10358 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.300390 10489 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:18.301111 16210 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:18.301465 10162 consensus_peers.cc:485] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f8b006a679a4b8d8d3551310b78fb3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:18.301460 15740 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:18.304347 16210 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:18.305012 10489 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:18.306208 10358 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:18.314394 9951 catalog_manager.cc:5484] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 4, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:18.323014 9950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_42"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:18.333087 9946 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:18.336616 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 90d885837d5e403890617f6ead1fd6c5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.336843 10337 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3effbb68eb4433bda2b2597ab1468a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.337144 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d885837d5e403890617f6ead1fd6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.336272 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 90d885837d5e403890617f6ead1fd6c5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.337718 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d885837d5e403890617f6ead1fd6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.337791 10337 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3effbb68eb4433bda2b2597ab1468a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.341445 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3effbb68eb4433bda2b2597ab1468a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.341984 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3effbb68eb4433bda2b2597ab1468a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.345407 10207 tablet_service.cc:1467] Processing CreateTablet for tablet 8c3effbb68eb4433bda2b2597ab1468a (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.345916 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3effbb68eb4433bda2b2597ab1468a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.356026 16404 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:18.357417 16404 tablet_bootstrap.cc:654] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.359998 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 90d885837d5e403890617f6ead1fd6c5 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.360649 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90d885837d5e403890617f6ead1fd6c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.363936 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 274d302b083b42019a6f03305fe516e7 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.364506 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274d302b083b42019a6f03305fe516e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.364912 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 274d302b083b42019a6f03305fe516e7 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.365375 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274d302b083b42019a6f03305fe516e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.370294 10206 tablet_service.cc:1467] Processing CreateTablet for tablet 274d302b083b42019a6f03305fe516e7 (DEFAULT_TABLE table=rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:18.370815 10206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274d302b083b42019a6f03305fe516e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.374568 16365 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:18.375053 16373 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:18.394097 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=af0cf6e68d134dd6aae9305f33205bc9] alter complete (version 3)
I20240503 02:13:18.396193 16366 tablet.cc:1722] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:18.397423 16409 tablet_bootstrap.cc:492] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:18.400807 16407 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:18.401857 16408 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:18.403589 16409 tablet_bootstrap.cc:654] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.403584 16408 tablet_bootstrap.cc:654] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.407740 16407 tablet_bootstrap.cc:654] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.410811 16408 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:18.411054 16408 ts_tablet_manager.cc:1367] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.000s
I20240503 02:13:18.412088 16408 raft_consensus.cc:348] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.412631 16408 raft_consensus.cc:374] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.412763 16408 raft_consensus.cc:729] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.413103 16408 consensus_queue.cc:260] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.413439 16409 tablet_bootstrap.cc:492] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:18.413628 16409 ts_tablet_manager.cc:1367] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20240503 02:13:18.414415 16409 raft_consensus.cc:348] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.414695 16409 raft_consensus.cc:374] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.414795 16409 raft_consensus.cc:729] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.415648 16408 ts_tablet_manager.cc:1398] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:18.417768 16407 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:18.421897 16407 ts_tablet_manager.cc:1367] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.000s
I20240503 02:13:18.423023 16407 raft_consensus.cc:348] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.423367 16407 raft_consensus.cc:374] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.423552 16407 raft_consensus.cc:729] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.423900 16407 consensus_queue.cc:260] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.426286 16407 ts_tablet_manager.cc:1398] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:18.415062 16409 consensus_queue.cc:260] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.433916 16409 ts_tablet_manager.cc:1398] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20240503 02:13:18.434429 16404 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:18.434613 16404 ts_tablet_manager.cc:1367] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.079s user 0.002s sys 0.000s
I20240503 02:13:18.435250 16404 raft_consensus.cc:348] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.435500 16404 raft_consensus.cc:374] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.435559 16404 raft_consensus.cc:729] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.435801 16404 consensus_queue.cc:260] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.436399 16404 ts_tablet_manager.cc:1398] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20240503 02:13:18.439210 16404 tablet_bootstrap.cc:492] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:18.442025 16404 tablet_bootstrap.cc:654] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.448900 10227 consensus_queue.cc:237] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:18.453871 10489 raft_consensus.cc:1261] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:18.454712 16297 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:18.457275 16404 tablet_bootstrap.cc:492] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:18.457535 16404 ts_tablet_manager.cc:1367] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20240503 02:13:18.458446 16404 raft_consensus.cc:348] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.458745 16404 raft_consensus.cc:374] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.458379 15740 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:18.458822 16404 raft_consensus.cc:729] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.459082 16404 consensus_queue.cc:260] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.459810 16404 ts_tablet_manager.cc:1398] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:18.359344 16406 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:18.460083 16408 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:18.461596 16408 tablet_bootstrap.cc:654] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.461745 16406 tablet_bootstrap.cc:654] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.465641 10489 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:18.471643 16415 tablet_bootstrap.cc:492] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:18.473696 16408 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:18.473726 16415 tablet_bootstrap.cc:654] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.474226 16408 ts_tablet_manager.cc:1367] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.000s
I20240503 02:13:18.474812 16406 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:18.474983 16406 ts_tablet_manager.cc:1367] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.116s user 0.004s sys 0.000s
I20240503 02:13:18.475003 16408 raft_consensus.cc:348] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.475219 16408 raft_consensus.cc:374] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.475311 16408 raft_consensus.cc:729] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.475562 16408 consensus_queue.cc:260] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.477026 16406 raft_consensus.cc:348] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.477326 16406 raft_consensus.cc:374] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.477401 16406 raft_consensus.cc:729] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.477732 16406 consensus_queue.cc:260] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.477900 16408 ts_tablet_manager.cc:1398] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20240503 02:13:18.478471 16406 ts_tablet_manager.cc:1398] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:18.479914 16406 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:18.481225 13040 raft_consensus.cc:1261] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:18.481626 16406 tablet_bootstrap.cc:654] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.482084 16297 consensus_queue.cc:1035] T be61de73e08c4998a51f02cafa46d421 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:18.483299 9950 catalog_manager.cc:5484] T be61de73e08c4998a51f02cafa46d421 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:18.491588 13040 raft_consensus.cc:2932] T be61de73e08c4998a51f02cafa46d421 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:18.496515 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet be61de73e08c4998a51f02cafa46d421 with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:18.496969 16315 tablet_replica.cc:330] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:18.497256 16315 raft_consensus.cc:2229] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:18.497411 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet be61de73e08c4998a51f02cafa46d421 with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:18.497546 16315 raft_consensus.cc:2258] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.498912 16315 ts_tablet_manager.cc:1875] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:18.505914 16315 ts_tablet_manager.cc:1888] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:18.506177 16315 log.cc:1196] T be61de73e08c4998a51f02cafa46d421 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/be61de73e08c4998a51f02cafa46d421
I20240503 02:13:18.507002 16415 tablet_bootstrap.cc:492] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:18.507014 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet be61de73e08c4998a51f02cafa46d421 (table rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]) successfully deleted
I20240503 02:13:18.512573 16415 ts_tablet_manager.cc:1367] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.004s
I20240503 02:13:18.513773 16415 raft_consensus.cc:348] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.520430 16415 raft_consensus.cc:374] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.520651 16415 raft_consensus.cc:729] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.520983 16415 consensus_queue.cc:260] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.521281 9946 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:18.521564 9946 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:18.521679 16415 ts_tablet_manager.cc:1398] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.009s user 0.001s sys 0.000s
I20240503 02:13:18.523013 16406 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:18.523209 16406 ts_tablet_manager.cc:1367] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20240503 02:13:18.524173 16406 raft_consensus.cc:348] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.524386 16406 raft_consensus.cc:374] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.524480 16406 raft_consensus.cc:729] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.524737 16406 consensus_queue.cc:260] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.525384 16406 ts_tablet_manager.cc:1398] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:18.525962 9946 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet e8c934099ef34318bff62c90eec41c07
I20240503 02:13:18.527060 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c934099ef34318bff62c90eec41c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:18 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:18.527581 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c934099ef34318bff62c90eec41c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:18 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:18.531069 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c934099ef34318bff62c90eec41c07 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:18 UTC) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:18.556210 16280 raft_consensus.cc:482] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.556509 16280 raft_consensus.cc:504] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.557511 16280 leader_election.cc:288] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:18.558084 16427 tablet_replica.cc:330] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:18.558472 16427 raft_consensus.cc:2229] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:18.558686 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d885837d5e403890617f6ead1fd6c5" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:18.559016 10091 raft_consensus.cc:2454] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:18.559157 16426 tablet_replica.cc:330] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:18.559423 16426 raft_consensus.cc:2229] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:18.560043 10294 leader_election.cc:302] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:13:18.560103 16432 tablet_replica.cc:330] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:18.560359 16432 raft_consensus.cc:2229] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:18.560642 16432 raft_consensus.cc:2258] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.561002 16280 raft_consensus.cc:2781] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.561123 16280 raft_consensus.cc:482] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.561214 16280 raft_consensus.cc:3037] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.563719 16432 ts_tablet_manager.cc:1875] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:18.564922 16427 raft_consensus.cc:2258] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.564844 16280 raft_consensus.cc:504] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.566255 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d885837d5e403890617f6ead1fd6c5" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:18.566483 10091 raft_consensus.cc:3037] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.569536 10091 raft_consensus.cc:2454] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:18.569569 16427 ts_tablet_manager.cc:1875] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:18.560740 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d885837d5e403890617f6ead1fd6c5" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:18.570204 10294 leader_election.cc:302] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f; no voters:
I20240503 02:13:18.570240 13040 raft_consensus.cc:2454] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:18.570654 16014 raft_consensus.cc:2781] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.570849 16426 raft_consensus.cc:2258] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:18.572139 16426 ts_tablet_manager.cc:1875] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:18.572077 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d885837d5e403890617f6ead1fd6c5" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:18.572362 13040 raft_consensus.cc:3037] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.578881 13040 raft_consensus.cc:2454] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:18.579264 16280 leader_election.cc:288] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:18.582960 16014 raft_consensus.cc:686] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:13:18.583312 16014 consensus_queue.cc:237] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:18.587210 16426 ts_tablet_manager.cc:1888] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:18.587484 16426 log.cc:1196] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/e8c934099ef34318bff62c90eec41c07
I20240503 02:13:18.588186 16426 ts_tablet_manager.cc:1909] T e8c934099ef34318bff62c90eec41c07 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:18.589617 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table_new_name [id=af0cf6e68d134dd6aae9305f33205bc9]) successfully deleted
I20240503 02:13:18.590101 16432 ts_tablet_manager.cc:1888] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:18.590250 16432 log.cc:1196] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/e8c934099ef34318bff62c90eec41c07
I20240503 02:13:18.590633 16432 ts_tablet_manager.cc:1909] T e8c934099ef34318bff62c90eec41c07 P eff8e8d11a4447218741bb994aa41082: Deleting consensus metadata
I20240503 02:13:18.590955 9946 catalog_manager.cc:5484] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:18.596055 16427 ts_tablet_manager.cc:1888] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:18.596276 16427 log.cc:1196] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/e8c934099ef34318bff62c90eec41c07
I20240503 02:13:18.596887 16427 ts_tablet_manager.cc:1909] T e8c934099ef34318bff62c90eec41c07 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:13:18.598798 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table_new_name [id=af0cf6e68d134dd6aae9305f33205bc9]) successfully deleted
I20240503 02:13:18.606791 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet e8c934099ef34318bff62c90eec41c07 (table rebalancer_dynamic_table_new_name [id=af0cf6e68d134dd6aae9305f33205bc9]) successfully deleted
I20240503 02:13:18.649966 9946 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:18.662379 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2e0cdbb96b42da92771c3242319231 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620]), partition=
I20240503 02:13:18.662473 13020 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2e0cdbb96b42da92771c3242319231 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620]), partition=
I20240503 02:13:18.662933 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2e0cdbb96b42da92771c3242319231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.662954 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2e0cdbb96b42da92771c3242319231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.663383 16014 raft_consensus.cc:482] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.663576 16014 raft_consensus.cc:504] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.664556 16014 leader_election.cc:288] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:18.664830 10338 tablet_service.cc:1467] Processing CreateTablet for tablet 2b2e0cdbb96b42da92771c3242319231 (DEFAULT_TABLE table=rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620]), partition=
I20240503 02:13:18.665237 10338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b2e0cdbb96b42da92771c3242319231. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.665946 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:18.666246 10489 raft_consensus.cc:2454] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:18.666992 10292 leader_election.cc:302] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:18.667299 16014 raft_consensus.cc:2781] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.667388 16014 raft_consensus.cc:482] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.667459 16014 raft_consensus.cc:3037] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.671411 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:18.671710 10227 raft_consensus.cc:2454] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 0.
I20240503 02:13:18.674834 16014 raft_consensus.cc:504] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.676416 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:18.676687 10227 raft_consensus.cc:3037] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.676565 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a" candidate_uuid: "47f901e617024baaa1865d12261ef84f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:18.676863 10489 raft_consensus.cc:3037] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.679982 10227 raft_consensus.cc:2454] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:18.680207 16014 leader_election.cc:288] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:18.680748 10489 raft_consensus.cc:2454] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47f901e617024baaa1865d12261ef84f in term 1.
I20240503 02:13:18.680824 10294 leader_election.cc:302] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:18.681671 16404 tablet_bootstrap.cc:492] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:18.682404 16415 tablet_bootstrap.cc:492] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:18.682482 16014 raft_consensus.cc:2781] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.682600 16014 raft_consensus.cc:686] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Leader. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:13:18.682883 16014 consensus_queue.cc:237] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.683773 16404 tablet_bootstrap.cc:654] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.684132 16415 tablet_bootstrap.cc:654] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.695924 16451 tablet_bootstrap.cc:492] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:18.701416 16404 tablet_bootstrap.cc:492] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: No bootstrap required, opened a new log
I20240503 02:13:18.701432 16415 tablet_bootstrap.cc:492] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:18.701632 16415 ts_tablet_manager.cc:1367] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.001s
I20240503 02:13:18.701802 16404 ts_tablet_manager.cc:1367] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.000s
I20240503 02:13:18.701870 16451 tablet_bootstrap.cc:654] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:18.701578 9946 catalog_manager.cc:5484] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f reported cstate change: term changed from 0 to 1, leader changed from <none> to 47f901e617024baaa1865d12261ef84f (127.4.5.67). New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:18.702739 16404 raft_consensus.cc:348] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.702764 16415 raft_consensus.cc:348] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.703004 16415 raft_consensus.cc:374] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.703102 16415 raft_consensus.cc:729] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.703238 16404 raft_consensus.cc:374] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.703366 16404 raft_consensus.cc:729] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.703477 16415 consensus_queue.cc:260] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.704135 16404 consensus_queue.cc:260] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.704478 16415 ts_tablet_manager.cc:1398] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20240503 02:13:18.704885 16404 ts_tablet_manager.cc:1398] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:18.732013 16451 tablet_bootstrap.cc:492] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:18.732235 16451 ts_tablet_manager.cc:1367] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.012s
I20240503 02:13:18.733222 16451 raft_consensus.cc:348] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.733760 16451 raft_consensus.cc:374] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:18.733918 16451 raft_consensus.cc:729] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:18.734238 16451 consensus_queue.cc:260] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.735137 16451 ts_tablet_manager.cc:1398] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:18.805459 16456 ts_tablet_manager.cc:902] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:18.808020 16456 tablet_copy_client.cc:308] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:18.809008 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 0a5125fbc8ec4ba887abce94a84efbdc from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:52743)
I20240503 02:13:18.809221 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 0a5125fbc8ec4ba887abce94a84efbdc from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:52743: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-0a5125fbc8ec4ba887abce94a84efbdc
I20240503 02:13:18.811875 16329 raft_consensus.cc:482] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.812103 16329 raft_consensus.cc:504] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.812775 10246 tablet_copy_source_session.cc:205] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:18.813457 16329 leader_election.cc:288] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:18.814224 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2e0cdbb96b42da92771c3242319231" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:18.814533 13040 raft_consensus.cc:2454] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:18.814828 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2e0cdbb96b42da92771c3242319231" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f" is_pre_election: true
I20240503 02:13:18.815039 10358 raft_consensus.cc:2454] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:18.815537 12843 leader_election.cc:302] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:18.816327 16329 raft_consensus.cc:2781] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.816480 16329 raft_consensus.cc:482] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.816570 16329 raft_consensus.cc:3037] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.818797 16329 raft_consensus.cc:504] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.819389 16329 leader_election.cc:288] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:18.819859 10358 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2e0cdbb96b42da92771c3242319231" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:18.819954 13040 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b2e0cdbb96b42da92771c3242319231" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:18.820065 10358 raft_consensus.cc:3037] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.820219 13040 raft_consensus.cc:3037] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.820371 16456 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0a5125fbc8ec4ba887abce94a84efbdc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.822387 10358 raft_consensus.cc:2454] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:18.822908 12842 leader_election.cc:302] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 47f901e617024baaa1865d12261ef84f, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
I20240503 02:13:18.823252 16329 raft_consensus.cc:2781] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.823427 16329 raft_consensus.cc:686] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:18.823421 13040 raft_consensus.cc:2454] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:18.823772 16329 consensus_queue.cc:237] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:18.829583 16456 tablet_copy_client.cc:783] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:18.829361 9950 catalog_manager.cc:5484] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:18.829896 16456 tablet_copy_client.cc:647] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:18.832506 16456 tablet_copy_client.cc:523] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:18.834954 16333 raft_consensus.cc:482] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:18.835144 16333 raft_consensus.cc:504] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.836125 16333 leader_election.cc:288] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:18.836952 13303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274d302b083b42019a6f03305fe516e7" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:18.837249 13303 raft_consensus.cc:2454] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:13:18.837193 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274d302b083b42019a6f03305fe516e7" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:18.836642 16456 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:18.837477 10227 raft_consensus.cc:2454] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:13:18.837829 10424 leader_election.cc:302] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:18.839169 16333 raft_consensus.cc:2781] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:18.839282 16333 raft_consensus.cc:482] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:18.839354 16333 raft_consensus.cc:3037] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.842552 16333 raft_consensus.cc:504] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.843891 13303 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274d302b083b42019a6f03305fe516e7" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:18.844225 13303 raft_consensus.cc:3037] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.847692 13303 raft_consensus.cc:2454] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:18.848178 10424 leader_election.cc:302] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:18.849000 16463 raft_consensus.cc:2781] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:18.849581 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "274d302b083b42019a6f03305fe516e7" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:18.849934 10227 raft_consensus.cc:3037] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:18.853411 16333 leader_election.cc:288] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:18.856181 10227 raft_consensus.cc:2454] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:18.860213 16463 raft_consensus.cc:686] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:18.860782 16463 consensus_queue.cc:237] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:18.866003 9950 catalog_manager.cc:5484] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:18.889214 16456 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:18.889686 16456 tablet_bootstrap.cc:492] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:18.889983 16456 ts_tablet_manager.cc:1367] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.053s user 0.045s sys 0.004s
I20240503 02:13:18.890898 16456 raft_consensus.cc:348] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:18.891211 16456 raft_consensus.cc:729] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:18.891458 16456 consensus_queue.cc:260] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:18.892112 16456 ts_tablet_manager.cc:1398] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:18.892618 16466 ts_tablet_manager.cc:902] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:18.892930 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-0a5125fbc8ec4ba887abce94a84efbdc received from {username='slave'} at 127.4.5.72:52743
I20240503 02:13:18.893071 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-0a5125fbc8ec4ba887abce94a84efbdc on tablet 0a5125fbc8ec4ba887abce94a84efbdc with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:18.901791 16466 tablet_copy_client.cc:308] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:18.903887 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 9f8b006a679a4b8d8d3551310b78fb3a from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:38111)
I20240503 02:13:18.904076 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 9f8b006a679a4b8d8d3551310b78fb3a from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:38111: session id = 8d454a3607d042dc9559ed59c3ccd84b-9f8b006a679a4b8d8d3551310b78fb3a
I20240503 02:13:18.907354 10246 tablet_copy_source_session.cc:205] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:18.911518 16466 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8b006a679a4b8d8d3551310b78fb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:18.918781 16466 tablet_copy_client.cc:783] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:18.919211 16466 tablet_copy_client.cc:647] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:18.921236 16466 tablet_copy_client.cc:523] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:18.924371 16466 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:18.935847 9950 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: DROP_COLUMN drop_column { name: "a" } } modify_external_catalogs: true
I20240503 02:13:18.963200 10358 raft_consensus.cc:1261] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:18.964340 16329 consensus_queue.cc:1035] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.965873 13040 raft_consensus.cc:1261] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:18.966550 16329 consensus_queue.cc:1035] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:18.995229 16014 consensus_queue.cc:1035] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.017933 9951 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 2b2e0cdbb96b42da92771c3242319231 (table rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620]). Expected version 1 got 0
I20240503 02:13:19.045914 16489 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:19.048045 16489 tablet.cc:1620] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:19.050565 9951 catalog_manager.cc:5576] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899) does not have the latest schema for tablet 2b2e0cdbb96b42da92771c3242319231 (table rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620]). Expected version 1 got 0
I20240503 02:13:19.062126 9946 catalog_manager.cc:5576] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471) does not have the latest schema for tablet 2b2e0cdbb96b42da92771c3242319231 (table rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620]). Expected version 1 got 0
I20240503 02:13:19.068907 16491 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:19.069635 16491 tablet.cc:1620] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: MemRowSet was empty: no flush needed.
I20240503 02:13:19.074644 9946 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620] alter complete (version 1)
I20240503 02:13:19.076835 9951 catalog_manager.cc:5576] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899) does not have the latest schema for tablet 2b2e0cdbb96b42da92771c3242319231 (table rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620]). Expected version 1 got 0
I20240503 02:13:19.081377 16490 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
11:a INT64 NULLABLE,
PRIMARY KEY (key)
) version 0 to (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1
I20240503 02:13:19.086705 16490 tablet.cc:1620] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:19.087391 9951 catalog_manager.cc:5576] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707) does not have the latest schema for tablet 2b2e0cdbb96b42da92771c3242319231 (table rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620]). Expected version 1 got 0
W20240503 02:13:19.098996 13107 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.69:33471, user_credentials={real_user=slave}} blocked reactor thread for 118398us
I20240503 02:13:19.125041 16466 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:19.126403 16466 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:19.127070 16466 ts_tablet_manager.cc:1367] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.203s user 0.054s sys 0.008s
I20240503 02:13:19.127923 16466 raft_consensus.cc:348] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:19.128274 16466 raft_consensus.cc:729] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:19.128542 16466 consensus_queue.cc:260] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:19.129374 16466 ts_tablet_manager.cc:1398] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:19.130892 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-9f8b006a679a4b8d8d3551310b78fb3a received from {username='slave'} at 127.4.5.70:38111
I20240503 02:13:19.131022 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-9f8b006a679a4b8d8d3551310b78fb3a on tablet 9f8b006a679a4b8d8d3551310b78fb3a with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:19.136873 16014 consensus_queue.cc:1035] T 90d885837d5e403890617f6ead1fd6c5 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.177471 9951 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } alter_schema_steps { type: ADD_COLUMN add_column { schema { name: "a" type: STRING is_nullable: true encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } } modify_external_catalogs: true
I20240503 02:13:19.186669 16489 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:19.187390 16489 tablet.cc:1620] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: MemRowSet was empty: no flush needed.
I20240503 02:13:19.187366 16490 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:19.187888 16490 tablet.cc:1620] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: MemRowSet was empty: no flush needed.
I20240503 02:13:19.187952 16491 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2
I20240503 02:13:19.188491 16491 tablet.cc:1620] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: MemRowSet was empty: no flush needed.
I20240503 02:13:19.201491 9950 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table [id=2e3b99e060ad4aa4ae2bcd598a0c4620] alter complete (version 2)
I20240503 02:13:19.213145 16014 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.218825 16014 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.253227 13302 raft_consensus.cc:1203] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:19.295167 16463 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.301505 16333 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:19.304674 9951 catalog_manager.cc:3330] Servicing AlterTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table" } new_table_name: "rebalancer_dynamic_table_new_name" modify_external_catalogs: true
I20240503 02:13:19.320859 16489 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:19.322574 16491 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:19.323305 16490 tablet.cc:1722] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Alter schema from (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 2 to (
10:key INT64 NOT NULL,
12:a STRING NULLABLE,
PRIMARY KEY (key)
) version 3
I20240503 02:13:19.334321 9952 catalog_manager.cc:6029] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: rebalancer_dynamic_table_new_name [id=2e3b99e060ad4aa4ae2bcd598a0c4620] alter complete (version 3)
I20240503 02:13:19.386569 9950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_extra_table_44"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "int_val"
type: INT32
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "string_val"
type: STRING
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 3
seed: 0
}
}
I20240503 02:13:19.399688 10470 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea10c9060e84b18817113eb8eae5409 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.399874 10074 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea10c9060e84b18817113eb8eae5409 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.400315 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea10c9060e84b18817113eb8eae5409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.400425 10074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea10c9060e84b18817113eb8eae5409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.403296 13153 tablet_service.cc:1467] Processing CreateTablet for tablet ac9acb6fb9c940ce83922596c849c4d8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.403998 13153 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9acb6fb9c940ce83922596c849c4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.405865 13290 tablet_service.cc:1467] Processing CreateTablet for tablet 7ea10c9060e84b18817113eb8eae5409 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.406491 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7ea10c9060e84b18817113eb8eae5409. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.415844 13152 tablet_service.cc:1467] Processing CreateTablet for tablet 587a3f3b0f7046d9a840d006c2f8741f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.416543 13152 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587a3f3b0f7046d9a840d006c2f8741f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.416384 10073 tablet_service.cc:1467] Processing CreateTablet for tablet ac9acb6fb9c940ce83922596c849c4d8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.416939 10073 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9acb6fb9c940ce83922596c849c4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.420127 10072 tablet_service.cc:1467] Processing CreateTablet for tablet 587a3f3b0f7046d9a840d006c2f8741f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.420733 10072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587a3f3b0f7046d9a840d006c2f8741f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.421660 10207 tablet_service.cc:1467] Processing CreateTablet for tablet ac9acb6fb9c940ce83922596c849c4d8 (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.422156 10207 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9acb6fb9c940ce83922596c849c4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.426016 12887 tablet_service.cc:1467] Processing CreateTablet for tablet 587a3f3b0f7046d9a840d006c2f8741f (DEFAULT_TABLE table=rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20240503 02:13:19.426704 12887 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587a3f3b0f7046d9a840d006c2f8741f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.438530 9951 catalog_manager.cc:2458] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:19.438858 9951 catalog_manager.cc:2642] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47904:
table { table_name: "rebalancer_dynamic_table_new_name" } modify_external_catalogs: true
I20240503 02:13:19.440707 16515 tablet_bootstrap.cc:492] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:19.441462 16517 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:19.446913 16517 tablet_bootstrap.cc:654] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.448458 9951 catalog_manager.cc:5771] T 00000000000000000000000000000000 P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Sending DeleteTablet for 3 replicas of tablet 2b2e0cdbb96b42da92771c3242319231
I20240503 02:13:19.449635 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2e0cdbb96b42da92771c3242319231 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:19 UTC) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:19.449955 16520 tablet_bootstrap.cc:492] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:19.452297 12886 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2e0cdbb96b42da92771c3242319231 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:19 UTC) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:19.453408 16523 tablet_replica.cc:330] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:19.453754 16523 raft_consensus.cc:2229] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Raft consensus shutting down.
I20240503 02:13:19.454139 16519 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:19.455451 16520 tablet_bootstrap.cc:654] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.456216 16515 tablet_bootstrap.cc:654] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.457072 16519 tablet_bootstrap.cc:654] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.458190 16523 raft_consensus.cc:2258] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:19.459780 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b2e0cdbb96b42da92771c3242319231 with delete_type TABLET_DATA_DELETED (Table deleted at 2024-05-03 02:13:19 UTC) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:19.461011 16518 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:19.462121 16523 ts_tablet_manager.cc:1875] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:19.465891 16518 tablet_bootstrap.cc:654] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.459530 16517 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78: No bootstrap required, opened a new log
I20240503 02:13:19.466516 16517 ts_tablet_manager.cc:1367] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.004s
I20240503 02:13:19.469401 16516 tablet_bootstrap.cc:492] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:19.470695 16517 raft_consensus.cc:348] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.471328 16517 raft_consensus.cc:374] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.471690 16517 raft_consensus.cc:729] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.472656 16516 tablet_bootstrap.cc:654] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.472242 16517 consensus_queue.cc:260] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.480406 16517 ts_tablet_manager.cc:1398] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.013s user 0.001s sys 0.002s
I20240503 02:13:19.488742 16518 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:19.489240 16518 ts_tablet_manager.cc:1367] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.006s
I20240503 02:13:19.490878 16518 raft_consensus.cc:348] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.497856 16518 raft_consensus.cc:374] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.498299 16518 raft_consensus.cc:729] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.499627 16525 tablet_replica.cc:330] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:19.499733 13039 raft_consensus.cc:1203] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.3->[1.4-1.4] Dedup: 1.4->[]
I20240503 02:13:19.499593 16518 consensus_queue.cc:260] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.499933 16525 raft_consensus.cc:2229] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:19.500190 16525 raft_consensus.cc:2258] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:19.502585 16518 ts_tablet_manager.cc:1398] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.013s user 0.000s sys 0.003s
I20240503 02:13:19.503171 16518 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:19.504164 16523 ts_tablet_manager.cc:1888] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:19.504678 16525 ts_tablet_manager.cc:1875] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20240503 02:13:19.505887 16518 tablet_bootstrap.cc:654] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.506402 16523 log.cc:1196] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/2b2e0cdbb96b42da92771c3242319231
I20240503 02:13:19.507174 16523 ts_tablet_manager.cc:1909] T 2b2e0cdbb96b42da92771c3242319231 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting consensus metadata
I20240503 02:13:19.508745 16520 tablet_bootstrap.cc:492] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:19.508936 16520 ts_tablet_manager.cc:1367] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.059s user 0.009s sys 0.000s
I20240503 02:13:19.509860 16520 raft_consensus.cc:348] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.510296 16520 raft_consensus.cc:374] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.510444 16520 raft_consensus.cc:729] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.510799 16520 consensus_queue.cc:260] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.513829 16520 ts_tablet_manager.cc:1398] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:19.521004 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 2b2e0cdbb96b42da92771c3242319231 (table rebalancer_dynamic_table_new_name [id=2e3b99e060ad4aa4ae2bcd598a0c4620]) successfully deleted
I20240503 02:13:19.530294 16525 ts_tablet_manager.cc:1888] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:19.530540 16525 log.cc:1196] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/2b2e0cdbb96b42da92771c3242319231
I20240503 02:13:19.531603 16525 ts_tablet_manager.cc:1909] T 2b2e0cdbb96b42da92771c3242319231 P 47f901e617024baaa1865d12261ef84f: Deleting consensus metadata
I20240503 02:13:19.535971 16333 raft_consensus.cc:482] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.536262 16333 raft_consensus.cc:504] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.537441 16333 leader_election.cc:288] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:19.539511 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea10c9060e84b18817113eb8eae5409" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
W20240503 02:13:19.540740 10424 leader_election.cc:341] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:19.543653 16518 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad: No bootstrap required, opened a new log
I20240503 02:13:19.547777 16518 ts_tablet_manager.cc:1367] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.012s
I20240503 02:13:19.568660 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 2b2e0cdbb96b42da92771c3242319231 (table rebalancer_dynamic_table_new_name [id=2e3b99e060ad4aa4ae2bcd598a0c4620]) successfully deleted
I20240503 02:13:19.569592 9950 catalog_manager.cc:1904] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47904:
name: "rebalancer_dynamic_table"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "a"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
}
}
I20240503 02:13:19.592728 16518 raft_consensus.cc:348] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.593099 16518 raft_consensus.cc:374] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.593201 16518 raft_consensus.cc:729] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.593551 16518 consensus_queue.cc:260] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.594168 16518 ts_tablet_manager.cc:1398] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.046s user 0.001s sys 0.000s
I20240503 02:13:19.606897 16480 raft_consensus.cc:482] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.607174 16480 raft_consensus.cc:504] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.608335 16480 leader_election.cc:288] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:19.609717 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9acb6fb9c940ce83922596c849c4d8" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
W20240503 02:13:19.610543 13109 leader_election.cc:341] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:19.637969 16317 raft_consensus.cc:482] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.638217 16317 raft_consensus.cc:504] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.639133 16317 leader_election.cc:288] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:19.640338 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587a3f3b0f7046d9a840d006c2f8741f" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:19.640775 13168 raft_consensus.cc:2454] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 0.
I20240503 02:13:19.641274 12843 leader_election.cc:302] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:19.641836 16317 raft_consensus.cc:2781] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.641968 16317 raft_consensus.cc:482] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.642060 16317 raft_consensus.cc:3037] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.648780 15740 raft_consensus.cc:1050] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:19.649227 16317 raft_consensus.cc:504] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.649853 15740 consensus_queue.cc:237] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:19.653144 16317 leader_election.cc:288] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:19.656633 13168 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "587a3f3b0f7046d9a840d006c2f8741f" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:19.656970 13168 raft_consensus.cc:3037] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.661718 13168 raft_consensus.cc:2454] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 1.
I20240503 02:13:19.667539 12843 leader_election.cc:302] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:19.670923 13019 tablet_service.cc:1467] Processing CreateTablet for tablet f385792ed7c8465eb7cd64142778c9bf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80fd16a658c24701a976069967996306]), partition=
I20240503 02:13:19.671186 10470 tablet_service.cc:1467] Processing CreateTablet for tablet f385792ed7c8465eb7cd64142778c9bf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80fd16a658c24701a976069967996306]), partition=
I20240503 02:13:19.671514 13019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f385792ed7c8465eb7cd64142778c9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.671626 10470 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f385792ed7c8465eb7cd64142778c9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.672175 13290 tablet_service.cc:1467] Processing CreateTablet for tablet f385792ed7c8465eb7cd64142778c9bf (DEFAULT_TABLE table=rebalancer_dynamic_table [id=80fd16a658c24701a976069967996306]), partition=
W20240503 02:13:19.672214 12842 leader_election.cc:341] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Not found: Tablet not found: 587a3f3b0f7046d9a840d006c2f8741f
I20240503 02:13:19.672777 13290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f385792ed7c8465eb7cd64142778c9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:19.677393 16329 raft_consensus.cc:2781] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.677660 16329 raft_consensus.cc:686] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:19.678031 16329 consensus_queue.cc:237] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.678755 16520 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
W20240503 02:13:19.680384 12842 leader_election.cc:341] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Not found: Tablet not found: 587a3f3b0f7046d9a840d006c2f8741f
I20240503 02:13:19.680593 16520 tablet_bootstrap.cc:654] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.689770 13305 raft_consensus.cc:1261] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:19.690990 16501 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:19.713186 10358 raft_consensus.cc:1261] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:13:19.713203 10026 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.69:33471, user_credentials={real_user=slave}} blocked reactor thread for 182725us
I20240503 02:13:19.717181 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea10c9060e84b18817113eb8eae5409" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
W20240503 02:13:19.717947 10426 leader_election.cc:341] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:19.718130 10426 leader_election.cc:302] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ec2f1283994a3089741715f811c850; no voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:19.718660 16333 raft_consensus.cc:2726] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:19.723417 9951 catalog_manager.cc:5484] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 0 to 1, leader changed from <none> to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:19.724678 16515 tablet_bootstrap.cc:492] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:19.724969 16515 ts_tablet_manager.cc:1367] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.284s user 0.006s sys 0.009s
I20240503 02:13:19.725628 16297 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:19.726975 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9acb6fb9c940ce83922596c849c4d8" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
W20240503 02:13:19.734390 13109 leader_election.cc:341] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Illegal state: must be running to vote when last-logged opid is not known
I20240503 02:13:19.734637 13109 leader_election.cc:302] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082
I20240503 02:13:19.735785 16480 raft_consensus.cc:2726] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20240503 02:13:19.740355 10489 raft_consensus.cc:1261] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:19.741952 16552 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:19.765690 16515 raft_consensus.cc:348] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.766351 16515 raft_consensus.cc:374] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.766566 16515 raft_consensus.cc:729] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.766964 16521 tablet_replica.cc:330] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:19.767344 16521 raft_consensus.cc:2229] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:19.767706 16521 raft_consensus.cc:2258] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:19.769901 16521 ts_tablet_manager.cc:1875] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20240503 02:13:19.780912 10161 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 46485.1us
W20240503 02:13:19.778811 10028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 153398us
I20240503 02:13:19.784611 16515 consensus_queue.cc:260] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.789238 16515 ts_tablet_manager.cc:1398] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.062s user 0.000s sys 0.003s
I20240503 02:13:19.790889 16515 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:19.794543 16520 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: No bootstrap required, opened a new log
I20240503 02:13:19.794927 16520 ts_tablet_manager.cc:1367] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.116s user 0.001s sys 0.011s
I20240503 02:13:19.796082 16515 tablet_bootstrap.cc:654] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.796053 16520 raft_consensus.cc:348] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:19.796505 16520 raft_consensus.cc:374] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.796634 16520 raft_consensus.cc:729] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.796993 16520 consensus_queue.cc:260] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:19.797972 16547 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:19.797633 16553 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:19.800271 16547 tablet_bootstrap.cc:654] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.804934 10358 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:19.811275 9952 catalog_manager.cc:5484] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:19.814631 16520 ts_tablet_manager.cc:1398] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.020s user 0.000s sys 0.002s
I20240503 02:13:19.817415 10491 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:19.818584 16519 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082: No bootstrap required, opened a new log
I20240503 02:13:19.818753 16519 ts_tablet_manager.cc:1367] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.365s user 0.061s sys 0.076s
I20240503 02:13:19.819528 16516 tablet_bootstrap.cc:492] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:19.819741 16516 ts_tablet_manager.cc:1367] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.351s user 0.019s sys 0.115s
I20240503 02:13:19.821533 16516 raft_consensus.cc:348] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.824429 16516 raft_consensus.cc:374] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.824723 16516 raft_consensus.cc:729] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.825173 16516 consensus_queue.cc:260] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.826190 16515 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:19.826462 16515 ts_tablet_manager.cc:1367] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.002s
I20240503 02:13:19.829439 16515 raft_consensus.cc:348] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.829967 16515 raft_consensus.cc:374] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.830116 16515 raft_consensus.cc:729] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.830519 16515 consensus_queue.cc:260] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.830557 16519 raft_consensus.cc:348] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.830827 16519 raft_consensus.cc:374] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.830953 16519 raft_consensus.cc:729] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.831210 16519 consensus_queue.cc:260] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.835397 16515 ts_tablet_manager.cc:1398] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:13:19.836216 16515 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:19.837116 13305 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:19.842384 16516 ts_tablet_manager.cc:1398] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20240503 02:13:19.845857 10227 consensus_queue.cc:237] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:19.852093 16521 ts_tablet_manager.cc:1888] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20240503 02:13:19.852311 16521 log.cc:1196] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/2b2e0cdbb96b42da92771c3242319231
I20240503 02:13:19.856564 16519 ts_tablet_manager.cc:1398] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.013s user 0.000s sys 0.001s
I20240503 02:13:19.858767 16516 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:19.858985 16515 tablet_bootstrap.cc:654] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.862700 13302 raft_consensus.cc:1261] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:19.865903 16547 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b: No bootstrap required, opened a new log
I20240503 02:13:19.866246 16547 ts_tablet_manager.cc:1367] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.000s
I20240503 02:13:19.867275 16547 raft_consensus.cc:348] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:19.867812 16547 raft_consensus.cc:374] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.868009 16547 raft_consensus.cc:729] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.869879 16516 tablet_bootstrap.cc:654] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Neither blocks nor log segments found. Creating new log.
I20240503 02:13:19.870730 16547 consensus_queue.cc:260] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:19.871806 16547 ts_tablet_manager.cc:1398] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:19.873512 10491 raft_consensus.cc:1261] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:19.883817 16521 ts_tablet_manager.cc:1909] T 2b2e0cdbb96b42da92771c3242319231 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting consensus metadata
I20240503 02:13:19.884414 16501 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:19.889075 16501 consensus_queue.cc:1035] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:19.894052 16515 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: No bootstrap required, opened a new log
I20240503 02:13:19.894243 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 2b2e0cdbb96b42da92771c3242319231 (table rebalancer_dynamic_table_new_name [id=2e3b99e060ad4aa4ae2bcd598a0c4620]) successfully deleted
I20240503 02:13:19.894788 16515 ts_tablet_manager.cc:1367] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.002s
I20240503 02:13:19.895741 16554 raft_consensus.cc:482] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.896062 16554 raft_consensus.cc:504] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.898011 16515 raft_consensus.cc:348] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.898238 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9acb6fb9c940ce83922596c849c4d8" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:19.899279 10091 raft_consensus.cc:2454] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:19.900223 10161 leader_election.cc:302] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:19.900889 16554 leader_election.cc:288] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:19.901062 16501 raft_consensus.cc:2781] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.901255 16501 raft_consensus.cc:482] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.901404 16501 raft_consensus.cc:3037] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.901722 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9acb6fb9c940ce83922596c849c4d8" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:19.902014 13166 raft_consensus.cc:2454] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 0.
I20240503 02:13:19.906895 16501 raft_consensus.cc:504] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.908604 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9acb6fb9c940ce83922596c849c4d8" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:19.908933 10091 raft_consensus.cc:3037] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.912288 10091 raft_consensus.cc:2454] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:19.913056 10161 leader_election.cc:302] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:19.913398 15740 raft_consensus.cc:2781] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.913631 15740 raft_consensus.cc:686] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:19.913936 15740 consensus_queue.cc:237] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } }
I20240503 02:13:19.914332 13166 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ac9acb6fb9c940ce83922596c849c4d8" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:19.914580 13166 raft_consensus.cc:3037] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.919780 13166 raft_consensus.cc:2454] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:19.921753 16501 leader_election.cc:288] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 1 election: Requested vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:19.922350 16515 raft_consensus.cc:374] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.923640 16515 raft_consensus.cc:729] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.924813 16515 consensus_queue.cc:260] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } }
I20240503 02:13:19.928853 16515 ts_tablet_manager.cc:1398] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.034s user 0.001s sys 0.002s
I20240503 02:13:19.932256 15740 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:19.934734 16516 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2: No bootstrap required, opened a new log
I20240503 02:13:19.935122 16516 ts_tablet_manager.cc:1367] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.076s user 0.008s sys 0.008s
I20240503 02:13:19.939903 9951 catalog_manager.cc:5484] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 0 to 1, leader changed from <none> to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:19.938031 16516 raft_consensus.cc:348] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:19.941589 16516 raft_consensus.cc:374] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20240503 02:13:19.941781 16516 raft_consensus.cc:729] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: FOLLOWER
I20240503 02:13:19.942176 16516 consensus_queue.cc:260] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:19.943266 16516 ts_tablet_manager.cc:1398] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20240503 02:13:19.952366 10491 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:19.956560 16531 raft_consensus.cc:482] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.956779 16531 raft_consensus.cc:504] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.957746 16531 leader_election.cc:288] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:19.958688 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea10c9060e84b18817113eb8eae5409" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:19.959000 10089 raft_consensus.cc:2454] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 0.
I20240503 02:13:19.959923 13245 leader_election.cc:302] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:13:19.960397 16531 raft_consensus.cc:2781] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.960548 16531 raft_consensus.cc:482] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.960639 16531 raft_consensus.cc:3037] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.963997 16531 raft_consensus.cc:504] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.965273 16531 leader_election.cc:288] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:19.965888 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea10c9060e84b18817113eb8eae5409" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:19.966212 10089 raft_consensus.cc:3037] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.969974 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea10c9060e84b18817113eb8eae5409" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:19.970263 10489 raft_consensus.cc:3037] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.973773 10489 raft_consensus.cc:2454] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:19.973981 9946 catalog_manager.cc:5484] T 0a5125fbc8ec4ba887abce94a84efbdc P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:19.975806 10089 raft_consensus.cc:2454] T 7ea10c9060e84b18817113eb8eae5409 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
I20240503 02:13:19.976706 13245 leader_election.cc:302] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:13:19.977422 16531 raft_consensus.cc:2781] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:19.977663 16531 raft_consensus.cc:686] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:19.978029 16531 consensus_queue.cc:237] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } }
I20240503 02:13:19.979813 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7ea10c9060e84b18817113eb8eae5409" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:19.980118 10491 raft_consensus.cc:2362] T 7ea10c9060e84b18817113eb8eae5409 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1. Re-sending same reply.
I20240503 02:13:19.980408 16560 raft_consensus.cc:482] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:19.980584 16560 raft_consensus.cc:504] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:19.981356 16560 leader_election.cc:288] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:19.983644 13305 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:19.984016 13305 raft_consensus.cc:2454] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:13:19.985108 10424 leader_election.cc:302] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:19.985558 16560 raft_consensus.cc:2781] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20240503 02:13:19.985646 16560 raft_consensus.cc:482] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:19.985704 16560 raft_consensus.cc:3037] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:19.989527 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet 0a5125fbc8ec4ba887abce94a84efbdc with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:19.990002 16525 tablet_replica.cc:330] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:19.990291 16525 raft_consensus.cc:2229] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:19.990672 16525 raft_consensus.cc:2258] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:19.991935 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:19.992389 13039 raft_consensus.cc:2454] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 0.
I20240503 02:13:19.992491 16525 ts_tablet_manager.cc:1875] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:19.993817 13305 raft_consensus.cc:2932] T 0a5125fbc8ec4ba887abce94a84efbdc P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:19.997401 16560 raft_consensus.cc:504] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:19.998981 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:19.999274 13302 raft_consensus.cc:3037] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.001080 16525 ts_tablet_manager.cc:1888] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:20.002838 13302 raft_consensus.cc:2454] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:20.004853 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0a5125fbc8ec4ba887abce94a84efbdc with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:20.000355 13039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:20.005645 13039 raft_consensus.cc:3037] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.005739 16525 log.cc:1196] T 0a5125fbc8ec4ba887abce94a84efbdc P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/0a5125fbc8ec4ba887abce94a84efbdc
I20240503 02:13:20.011001 9947 catalog_manager.cc:5484] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:20.016211 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 0a5125fbc8ec4ba887abce94a84efbdc (table rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]) successfully deleted
I20240503 02:13:20.021539 16501 raft_consensus.cc:1050] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:20.022611 16501 consensus_queue.cc:237] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:20.023294 13039 raft_consensus.cc:2454] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:19.998153 16560 leader_election.cc:288] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:20.007186 10424 leader_election.cc:302] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:20.027665 13040 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.027910 10358 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.028088 16563 raft_consensus.cc:2781] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader election won for term 1
I20240503 02:13:20.028245 16563 raft_consensus.cc:686] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:20.028643 16563 consensus_queue.cc:237] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } }
I20240503 02:13:20.028723 16297 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.035112 16297 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:20.044591 10491 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:20.073786 16501 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.014s
I20240503 02:13:20.078440 9947 catalog_manager.cc:5484] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 0 to 1, leader changed from <none> to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:20.099481 16297 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:20.102813 13039 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:20.106154 10358 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:20.114975 16549 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.122462 9946 catalog_manager.cc:5484] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:20.131512 10489 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:20.139784 10089 raft_consensus.cc:3037] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 0 FOLLOWER]: Advancing to term 1
I20240503 02:13:20.151800 10224 consensus_queue.cc:237] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:20.154472 10491 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.159435 16297 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.185411 13040 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:20.187151 16501 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:20.198835 16297 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:20.199437 16549 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.201184 13040 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:20.203182 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9f8b006a679a4b8d8d3551310b78fb3a with cas_config_opid_index 5: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:20.205067 10491 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:20.207737 9947 catalog_manager.cc:5484] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 5 to 6, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:20.213937 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8b006a679a4b8d8d3551310b78fb3a with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 6) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:20.214560 16525 tablet_replica.cc:330] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:20.214802 16525 raft_consensus.cc:2229] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:20.215140 16525 raft_consensus.cc:2258] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:20.216364 16525 ts_tablet_manager.cc:1875] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:20.228634 16525 ts_tablet_manager.cc:1888] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.5
I20240503 02:13:20.229051 16525 log.cc:1196] T 9f8b006a679a4b8d8d3551310b78fb3a P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/9f8b006a679a4b8d8d3551310b78fb3a
I20240503 02:13:20.229955 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 9f8b006a679a4b8d8d3551310b78fb3a (table rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]) successfully deleted
I20240503 02:13:20.389742 15740 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.415935 16501 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.484772 16531 consensus_queue.cc:1035] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.504809 16214 consensus_queue.cc:1035] T 7ea10c9060e84b18817113eb8eae5409 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.588980 16563 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.614950 16333 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20240503 02:13:20.916416 10072 tablet_service.cc:1429] Tablet server has 16 leaders and 0 scanners
I20240503 02:13:20.918390 13019 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:13:20.925086 10338 tablet_service.cc:1429] Tablet server has 14 leaders and 0 scanners
I20240503 02:13:20.928287 13290 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:13:20.936435 10470 tablet_service.cc:1429] Tablet server has 10 leaders and 0 scanners
I20240503 02:13:21.058234 12887 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
I20240503 02:13:21.066867 13153 tablet_service.cc:1429] Tablet server has 8 leaders and 0 scanners
I20240503 02:13:21.132593 10207 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:13:21.307593 10224 consensus_queue.cc:237] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:21.310190 10089 raft_consensus.cc:1261] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.311581 12908 raft_consensus.cc:1261] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:21.311955 10162 consensus_peers.cc:485] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ccd3faa7afbb4b92a4a4c4645e477771. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.312304 16297 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.314451 16297 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.316764 16501 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:21.317512 12908 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:21.325546 9943 catalog_manager.cc:5484] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:21.337698 10089 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:21.367537 13166 consensus_queue.cc:237] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 2.1, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:21.370697 12908 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
W20240503 02:13:21.370782 13109 consensus_peers.cc:485] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 004a7ce056044d25ab68a44e51688ca3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.371853 10489 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 2 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:13:21.371958 16671 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.373029 16671 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.379170 16671 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 2 LEADER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:21.379433 12907 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:21.385636 10489 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Committing config change with OpId 2.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:21.388077 9947 catalog_manager.cc:5484] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 2 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:21.435642 10359 consensus_queue.cc:237] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
W20240503 02:13:21.439108 10292 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d2d4b31cd7334ed6b18288c1b6d3f958. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.440544 12907 raft_consensus.cc:1261] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.440696 13040 raft_consensus.cc:1261] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.441388 16433 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.441717 16280 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.445026 16433 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:21.445593 13040 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:21.460669 9950 catalog_manager.cc:5484] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:21.469903 12907 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:13:21.520493 13305 consensus_queue.cc:237] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
W20240503 02:13:21.523533 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2a328e790e844b5aa9a1cf0fa16ee8c. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.523612 12907 raft_consensus.cc:1261] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.524155 13166 raft_consensus.cc:1261] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.524300 16619 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.001s
I20240503 02:13:21.525003 16214 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.535878 16214 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:21.542557 13166 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:21.555131 9947 catalog_manager.cc:5484] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:21.563431 12908 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:21.591977 12908 consensus_queue.cc:237] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:21.594938 10089 raft_consensus.cc:1261] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:21.595237 12843 consensus_peers.cc:485] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 587a3f3b0f7046d9a840d006c2f8741f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.595456 13166 raft_consensus.cc:1261] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.596802 16651 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.597385 16317 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.600557 16317 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:21.604159 10089 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:21.615866 9947 catalog_manager.cc:5484] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:21.618507 13166 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:21.641686 13166 consensus_queue.cc:237] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:21.645185 10489 raft_consensus.cc:1261] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.645390 12908 raft_consensus.cc:1261] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.646006 16671 consensus_queue.cc:1035] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.646464 16695 consensus_queue.cc:1035] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:21.646821 13109 consensus_peers.cc:485] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0f6056d9c9a949a6907510bd619825f6. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.649482 16695 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:21.650086 10489 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:21.650703 12908 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:21.658341 9950 catalog_manager.cc:5484] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:21.682924 13040 consensus_queue.cc:237] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:21.687377 10489 raft_consensus.cc:1261] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.688055 16316 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.693459 16316 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:21.698804 10491 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:21.708168 9952 catalog_manager.cc:5484] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
W20240503 02:13:21.709296 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09e0373a72654a1082464fac8d428483. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.722900 12908 raft_consensus.cc:1261] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.724077 16747 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.741533 12908 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:21.790498 16757 ts_tablet_manager.cc:902] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:21.824111 16757 tablet_copy_client.cc:308] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.71:46557
I20240503 02:13:21.826041 13195 tablet_copy_service.cc:140] P f93b103af083471b8e8c0376e9148aad: Received BeginTabletCopySession request for tablet 004a7ce056044d25ab68a44e51688ca3 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:41557)
I20240503 02:13:21.826279 13195 tablet_copy_service.cc:161] P f93b103af083471b8e8c0376e9148aad: Beginning new tablet copy session on tablet 004a7ce056044d25ab68a44e51688ca3 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:41557: session id = eff8e8d11a4447218741bb994aa41082-004a7ce056044d25ab68a44e51688ca3
I20240503 02:13:21.829488 13195 tablet_copy_source_session.cc:205] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:21.831645 16757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 004a7ce056044d25ab68a44e51688ca3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:21.832010 10224 consensus_queue.cc:237] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:21.839692 16757 tablet_copy_client.cc:783] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:21.840029 16757 tablet_copy_client.cc:647] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:21.840874 13305 raft_consensus.cc:1261] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.841535 16501 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.841918 16757 tablet_copy_client.cc:523] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:13:21.842381 10161 consensus_peers.cc:485] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0df3294fa2d74baea8c573a5dcfa96bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.844805 12908 raft_consensus.cc:1261] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:21.845036 16501 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:21.845855 16757 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:21.849175 16297 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:21.849737 13305 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:21.868746 9943 catalog_manager.cc:5484] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:21.869874 12907 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:21.879052 16757 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:21.879550 16757 tablet_bootstrap.cc:492] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:21.879870 16757 ts_tablet_manager.cc:1367] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.034s user 0.020s sys 0.008s
I20240503 02:13:21.880755 16757 raft_consensus.cc:348] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:21.881014 16757 raft_consensus.cc:729] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:21.881191 16757 consensus_queue.cc:260] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 2.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:21.883225 16757 ts_tablet_manager.cc:1398] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20240503 02:13:21.884395 13195 tablet_copy_service.cc:342] P f93b103af083471b8e8c0376e9148aad: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-004a7ce056044d25ab68a44e51688ca3 received from {username='slave'} at 127.4.5.66:41557
I20240503 02:13:21.884624 13195 tablet_copy_service.cc:433] P f93b103af083471b8e8c0376e9148aad: ending tablet copy session eff8e8d11a4447218741bb994aa41082-004a7ce056044d25ab68a44e51688ca3 on tablet 004a7ce056044d25ab68a44e51688ca3 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:21.903486 10359 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:21.907109 13166 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:21.907536 12908 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20240503 02:13:21.908298 16707 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:21.909516 16433 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20240503 02:13:21.910672 10292 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: db2da39193164d1ebf830282258d47d7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:21.913031 16280 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 2 LEADER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:21.913738 13166 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:21.920048 9952 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 5 to 6, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 2 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:21.943835 12907 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.6: config changed from index 5 to 6, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:21.957751 16777 ts_tablet_manager.cc:902] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:21.974344 16781 ts_tablet_manager.cc:902] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:21.988986 16777 tablet_copy_client.cc:308] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:21.989912 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet ccd3faa7afbb4b92a4a4c4645e477771 from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:38111)
I20240503 02:13:21.990126 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet ccd3faa7afbb4b92a4a4c4645e477771 from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:38111: session id = 8d454a3607d042dc9559ed59c3ccd84b-ccd3faa7afbb4b92a4a4c4645e477771
I20240503 02:13:21.991612 12907 consensus_queue.cc:237] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:21.995721 10246 tablet_copy_source_session.cc:205] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.001132 16777 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ccd3faa7afbb4b92a4a4c4645e477771. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.012987 16777 tablet_copy_client.cc:783] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
W20240503 02:13:22.013325 12842 consensus_peers.cc:485] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 90b2d976f9fe42d2882b1c1ef7a01eb3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.013600 16777 tablet_copy_client.cc:647] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.014813 13305 raft_consensus.cc:1261] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.015506 13040 raft_consensus.cc:1261] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.016333 16317 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.019181 16317 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.023550 16317 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:22.024718 16781 tablet_copy_client.cc:308] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:22.025689 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet d2d4b31cd7334ed6b18288c1b6d3f958 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:39119)
I20240503 02:13:22.025916 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet d2d4b31cd7334ed6b18288c1b6d3f958 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:39119: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-d2d4b31cd7334ed6b18288c1b6d3f958
I20240503 02:13:22.025450 13303 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:22.028380 16777 tablet_copy_client.cc:523] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.029191 10380 tablet_copy_source_session.cc:205] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.031760 16781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d2d4b31cd7334ed6b18288c1b6d3f958. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.039755 9952 catalog_manager.cc:5484] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:22.045343 16777 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:22.048743 16796 ts_tablet_manager.cc:902] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:22.059679 13040 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:22.073908 16781 tablet_copy_client.cc:783] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.074277 16781 tablet_copy_client.cc:647] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.079758 16781 tablet_copy_client.cc:523] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.086769 16777 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.087318 16777 tablet_bootstrap.cc:492] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:22.087626 16777 ts_tablet_manager.cc:1367] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.005s
I20240503 02:13:22.088372 16781 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap starting.
I20240503 02:13:22.094700 16777 raft_consensus.cc:348] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:22.095043 16777 raft_consensus.cc:729] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:22.095328 16777 consensus_queue.cc:260] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:22.098754 16777 ts_tablet_manager.cc:1398] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.011s user 0.003s sys 0.003s
I20240503 02:13:22.099536 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-ccd3faa7afbb4b92a4a4c4645e477771 received from {username='slave'} at 127.4.5.70:38111
I20240503 02:13:22.099694 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-ccd3faa7afbb4b92a4a4c4645e477771 on tablet ccd3faa7afbb4b92a4a4c4645e477771 with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:22.109467 16796 tablet_copy_client.cc:308] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:22.112684 16781 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.113489 16781 tablet_bootstrap.cc:492] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Bootstrap complete.
I20240503 02:13:22.113895 16781 ts_tablet_manager.cc:1367] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.015s
I20240503 02:13:22.115295 16781 raft_consensus.cc:348] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:22.115650 16781 raft_consensus.cc:729] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Initialized, Role: LEARNER
I20240503 02:13:22.115957 16781 consensus_queue.cc:260] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:13:22.116926 16781 ts_tablet_manager.cc:1398] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:22.119968 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-d2d4b31cd7334ed6b18288c1b6d3f958 received from {username='slave'} at 127.4.5.72:39119
I20240503 02:13:22.120136 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session 8ee3fd5ad3624ea1b9f681d24e9aabf2-d2d4b31cd7334ed6b18288c1b6d3f958 on tablet d2d4b31cd7334ed6b18288c1b6d3f958 with peer 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:22.122125 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet 587a3f3b0f7046d9a840d006c2f8741f from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:39177)
I20240503 02:13:22.122357 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet 587a3f3b0f7046d9a840d006c2f8741f from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:39177: session id = 8d454a3607d042dc9559ed59c3ccd84b-587a3f3b0f7046d9a840d006c2f8741f
I20240503 02:13:22.125631 12928 tablet_copy_source_session.cc:205] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.130107 16796 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 587a3f3b0f7046d9a840d006c2f8741f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.137370 16796 tablet_copy_client.cc:783] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.137730 16796 tablet_copy_client.cc:647] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.140065 16796 tablet_copy_client.cc:523] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.147150 16809 ts_tablet_manager.cc:902] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:22.148036 16796 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:22.151690 10359 consensus_queue.cc:237] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:22.157775 16810 ts_tablet_manager.cc:902] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:22.160195 16757 ts_tablet_manager.cc:902] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
W20240503 02:13:22.165417 10292 consensus_peers.cc:485] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8c3effbb68eb4433bda2b2597ab1468a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.167219 10491 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.168624 10224 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.169153 16707 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.169590 16280 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.178570 16810 tablet_copy_client.cc:308] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.72:33671
I20240503 02:13:22.178570 16809 tablet_copy_client.cc:308] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:22.180732 13330 tablet_copy_service.cc:140] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Received BeginTabletCopySession request for tablet c2a328e790e844b5aa9a1cf0fa16ee8c from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:38343)
I20240503 02:13:22.180970 13330 tablet_copy_service.cc:161] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Beginning new tablet copy session on tablet c2a328e790e844b5aa9a1cf0fa16ee8c from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:38343: session id = 47f901e617024baaa1865d12261ef84f-c2a328e790e844b5aa9a1cf0fa16ee8c
I20240503 02:13:22.183128 16757 tablet_copy_client.cc:308] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.71:46557
I20240503 02:13:22.184104 13195 tablet_copy_service.cc:140] P f93b103af083471b8e8c0376e9148aad: Received BeginTabletCopySession request for tablet 0f6056d9c9a949a6907510bd619825f6 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:41557)
I20240503 02:13:22.184310 13195 tablet_copy_service.cc:161] P f93b103af083471b8e8c0376e9148aad: Beginning new tablet copy session on tablet 0f6056d9c9a949a6907510bd619825f6 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:41557: session id = eff8e8d11a4447218741bb994aa41082-0f6056d9c9a949a6907510bd619825f6
I20240503 02:13:22.185058 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet 09e0373a72654a1082464fac8d428483 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:42943)
I20240503 02:13:22.185284 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet 09e0373a72654a1082464fac8d428483 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:42943: session id = 47f901e617024baaa1865d12261ef84f-09e0373a72654a1082464fac8d428483
I20240503 02:13:22.187773 13195 tablet_copy_source_session.cc:205] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.190865 16757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0f6056d9c9a949a6907510bd619825f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.191216 16796 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.194844 13061 tablet_copy_source_session.cc:205] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.195905 13330 tablet_copy_source_session.cc:205] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.196404 16809 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09e0373a72654a1082464fac8d428483. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.198275 16707 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:22.199549 16757 tablet_copy_client.cc:783] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.199865 16757 tablet_copy_client.cc:647] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.201910 16810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c2a328e790e844b5aa9a1cf0fa16ee8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.202606 16757 tablet_copy_client.cc:523] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.202906 10224 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:22.205147 16809 tablet_copy_client.cc:783] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.205492 16809 tablet_copy_client.cc:647] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.207054 16757 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:22.209017 16796 tablet_bootstrap.cc:492] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:22.209453 16796 ts_tablet_manager.cc:1367] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.062s user 0.026s sys 0.009s
I20240503 02:13:22.210532 16796 raft_consensus.cc:348] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:22.210276 10491 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:22.210804 16796 raft_consensus.cc:729] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:22.211048 16796 consensus_queue.cc:260] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:22.211762 10227 raft_consensus.cc:1203] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.1->[2.2-2.2] Dedup: 2.2->[]
I20240503 02:13:22.213179 16796 ts_tablet_manager.cc:1398] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20240503 02:13:22.216316 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-587a3f3b0f7046d9a840d006c2f8741f received from {username='slave'} at 127.4.5.70:39177
I20240503 02:13:22.216568 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-587a3f3b0f7046d9a840d006c2f8741f on tablet 587a3f3b0f7046d9a840d006c2f8741f with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:22.221392 16757 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.221807 16757 tablet_bootstrap.cc:492] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:22.222237 16757 ts_tablet_manager.cc:1367] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.011s
I20240503 02:13:22.223115 16757 raft_consensus.cc:348] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:22.223349 16757 raft_consensus.cc:729] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:22.223546 16757 consensus_queue.cc:260] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:22.224531 16757 ts_tablet_manager.cc:1398] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:22.226756 13195 tablet_copy_service.cc:342] P f93b103af083471b8e8c0376e9148aad: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-0f6056d9c9a949a6907510bd619825f6 received from {username='slave'} at 127.4.5.66:41557
I20240503 02:13:22.226934 13195 tablet_copy_service.cc:433] P f93b103af083471b8e8c0376e9148aad: ending tablet copy session eff8e8d11a4447218741bb994aa41082-0f6056d9c9a949a6907510bd619825f6 on tablet 0f6056d9c9a949a6907510bd619825f6 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:22.230281 9952 catalog_manager.cc:5484] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 2, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:22.232734 16809 tablet_copy_client.cc:523] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.239039 16809 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:22.248596 16810 tablet_copy_client.cc:783] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.248939 16810 tablet_copy_client.cc:647] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.252171 16810 tablet_copy_client.cc:523] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.258157 16809 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.258666 16809 tablet_bootstrap.cc:492] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:22.258953 16809 ts_tablet_manager.cc:1367] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.020s user 0.012s sys 0.001s
I20240503 02:13:22.259385 16810 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:22.259869 16809 raft_consensus.cc:348] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:22.260125 16809 raft_consensus.cc:729] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:22.260347 16809 consensus_queue.cc:260] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:22.261040 16809 ts_tablet_manager.cc:1398] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20240503 02:13:22.262770 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-09e0373a72654a1082464fac8d428483 received from {username='slave'} at 127.4.5.67:42943
I20240503 02:13:22.263015 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session 47f901e617024baaa1865d12261ef84f-09e0373a72654a1082464fac8d428483 on tablet 09e0373a72654a1082464fac8d428483 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:22.271348 16810 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.271986 16810 tablet_bootstrap.cc:492] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:22.272281 16810 ts_tablet_manager.cc:1367] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.013s user 0.013s sys 0.000s
I20240503 02:13:22.273072 16810 raft_consensus.cc:348] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:22.273289 16810 raft_consensus.cc:729] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:22.273540 16810 consensus_queue.cc:260] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:22.274075 16810 ts_tablet_manager.cc:1398] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20240503 02:13:22.274946 13330 tablet_copy_service.cc:342] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-c2a328e790e844b5aa9a1cf0fa16ee8c received from {username='slave'} at 127.4.5.67:38343
I20240503 02:13:22.275125 13330 tablet_copy_service.cc:433] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: ending tablet copy session 47f901e617024baaa1865d12261ef84f-c2a328e790e844b5aa9a1cf0fa16ee8c on tablet c2a328e790e844b5aa9a1cf0fa16ee8c with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:22.301008 13168 consensus_queue.cc:237] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:22.304280 13303 raft_consensus.cc:1261] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:22.305248 13109 consensus_peers.cc:485] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 918c524462ad41008d2f54f8e954e9bd. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.305953 12908 raft_consensus.cc:1261] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.310907 16671 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.311414 15059 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.318225 16671 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:22.318805 13303 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:22.326861 12908 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:22.330339 9952 catalog_manager.cc:5484] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:22.358119 13040 consensus_queue.cc:237] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:22.362651 10359 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.363276 16750 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:22.363734 12973 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 317ce62718cb412bbf04569cab14309f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.363718 13166 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:22.364282 16750 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:22.366787 16754 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:22.367666 10359 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:22.369457 13166 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:22.374874 9943 catalog_manager.cc:5484] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 4 to 5, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.401724 13040 raft_consensus.cc:1203] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:22.403486 13303 raft_consensus.cc:1203] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:22.422271 13303 consensus_queue.cc:237] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
W20240503 02:13:22.425966 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09468c67719244c18d200463892a4f84. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.426987 13040 raft_consensus.cc:1261] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.428331 16214 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.429999 12908 raft_consensus.cc:1261] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.431556 16719 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.436062 16720 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:22.437292 12908 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:22.445143 9952 catalog_manager.cc:5484] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:22.449744 13040 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:22.451246 16859 ts_tablet_manager.cc:902] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:22.452971 13038 raft_consensus.cc:1203] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:22.478929 16859 tablet_copy_client.cc:308] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:22.480345 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 0df3294fa2d74baea8c573a5dcfa96bd from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:44821)
I20240503 02:13:22.480633 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 0df3294fa2d74baea8c573a5dcfa96bd from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:44821: session id = 031a0afb01b6469299fa829fa3467ef7-0df3294fa2d74baea8c573a5dcfa96bd
I20240503 02:13:22.484383 10246 tablet_copy_source_session.cc:205] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.485997 16859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0df3294fa2d74baea8c573a5dcfa96bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.493201 16859 tablet_copy_client.cc:783] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.493554 16859 tablet_copy_client.cc:647] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.495478 10224 consensus_queue.cc:237] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
W20240503 02:13:22.502458 10159 consensus_peers.cc:485] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9f8b006a679a4b8d8d3551310b78fb3a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.502712 13039 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:22.503266 10491 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:22.503861 16765 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:22.507055 16297 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:22.509506 16859 tablet_copy_client.cc:523] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.514235 16859 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:22.520918 16873 ts_tablet_manager.cc:902] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:22.522893 16765 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:22.530247 10489 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:22.537425 16873 tablet_copy_client.cc:308] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:22.538511 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet db2da39193164d1ebf830282258d47d7 from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:34297)
I20240503 02:13:22.538748 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet db2da39193164d1ebf830282258d47d7 from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:34297: session id = c6ec2f1283994a3089741715f811c850-db2da39193164d1ebf830282258d47d7
I20240503 02:13:22.541823 10380 tablet_copy_source_session.cc:205] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.544836 16873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet db2da39193164d1ebf830282258d47d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.551621 9950 catalog_manager.cc:5484] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:22.552414 16873 tablet_copy_client.cc:783] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.552747 16873 tablet_copy_client.cc:647] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.557114 10224 raft_consensus.cc:1203] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:22.559026 16873 tablet_copy_client.cc:523] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.569128 16873 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:22.570390 16757 ts_tablet_manager.cc:902] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:22.560869 16859 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.588788 13040 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:22.592586 16859 tablet_bootstrap.cc:492] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:22.597133 16859 ts_tablet_manager.cc:1367] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.083s user 0.008s sys 0.018s
I20240503 02:13:22.600126 16859 raft_consensus.cc:348] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:22.603614 16859 raft_consensus.cc:729] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:22.604434 16859 consensus_queue.cc:260] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:22.609931 16859 ts_tablet_manager.cc:1398] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20240503 02:13:22.615731 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-0df3294fa2d74baea8c573a5dcfa96bd received from {username='slave'} at 127.4.5.65:44821
I20240503 02:13:22.615924 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-0df3294fa2d74baea8c573a5dcfa96bd on tablet 0df3294fa2d74baea8c573a5dcfa96bd with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:22.616390 16757 tablet_copy_client.cc:308] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:22.617208 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:52841)
I20240503 02:13:22.617442 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:52841: session id = eff8e8d11a4447218741bb994aa41082-90b2d976f9fe42d2882b1c1ef7a01eb3
I20240503 02:13:22.619758 12928 tablet_copy_source_session.cc:205] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.622053 16757 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.634287 16757 tablet_copy_client.cc:783] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.634641 16757 tablet_copy_client.cc:647] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.636638 16757 tablet_copy_client.cc:523] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.638587 16873 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.639212 16873 tablet_bootstrap.cc:492] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:22.641280 16873 ts_tablet_manager.cc:1367] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.072s user 0.020s sys 0.031s
I20240503 02:13:22.642239 16873 raft_consensus.cc:348] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:22.642503 16873 raft_consensus.cc:729] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:22.642724 16873 consensus_queue.cc:260] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 2.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:22.643582 16873 ts_tablet_manager.cc:1398] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:22.645361 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-db2da39193164d1ebf830282258d47d7 received from {username='slave'} at 127.4.5.68:34297
I20240503 02:13:22.645507 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session c6ec2f1283994a3089741715f811c850-db2da39193164d1ebf830282258d47d7 on tablet db2da39193164d1ebf830282258d47d7 with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:22.649020 12908 consensus_queue.cc:237] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:22.651227 16880 raft_consensus.cc:1050] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:22.652184 16880 consensus_queue.cc:237] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 2.2, Last appended by leader: 0, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:22.657797 10224 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 [term 2 LEARNER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:22.658077 12907 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:22.658488 16757 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:22.659307 15059 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:22.660116 15059 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:22.664332 10358 raft_consensus.cc:1203] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
W20240503 02:13:22.666126 12840 consensus_peers.cc:485] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 189d7bee0d6f40a69f25aa578f3d1dec. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:22.667778 10227 raft_consensus.cc:1261] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.668548 16731 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.669992 13305 raft_consensus.cc:1261] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:22.676007 16731 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:22.683943 10489 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 2 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20240503 02:13:22.684652 16671 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:22.686090 16757 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.686523 16757 tablet_bootstrap.cc:492] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:22.686815 16757 ts_tablet_manager.cc:1367] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.028s user 0.011s sys 0.009s
I20240503 02:13:22.692607 10358 raft_consensus.cc:1203] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:22.696285 16757 raft_consensus.cc:348] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:22.696786 16757 raft_consensus.cc:729] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:22.697084 16757 consensus_queue.cc:260] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:22.697937 16757 ts_tablet_manager.cc:1398] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20240503 02:13:22.701565 16899 ts_tablet_manager.cc:902] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:22.702711 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-90b2d976f9fe42d2882b1c1ef7a01eb3 received from {username='slave'} at 127.4.5.66:52841
I20240503 02:13:22.702879 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session eff8e8d11a4447218741bb994aa41082-90b2d976f9fe42d2882b1c1ef7a01eb3 on tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:22.711129 16730 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:22.713433 16880 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 2 LEADER]: Committing config change with OpId 2.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.720930 10489 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.727190 10227 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:22.739182 13303 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:22.739573 9943 catalog_manager.cc:5484] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:22.750306 12908 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.770339 10227 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Committing config change with OpId 2.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.777237 9947 catalog_manager.cc:5484] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.793874 13166 consensus_queue.cc:237] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 2.3, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:22.797286 10224 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:22.798269 16906 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:22.799714 12908 raft_consensus.cc:1261] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 2 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:22.800273 16906 consensus_queue.cc:1035] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:22.815179 16880 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P f93b103af083471b8e8c0376e9148aad [term 2 LEADER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.818662 10224 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.819763 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 004a7ce056044d25ab68a44e51688ca3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:22.820088 16899 tablet_copy_client.cc:308] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:22.821177 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 8c3effbb68eb4433bda2b2597ab1468a from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:56263)
I20240503 02:13:22.821367 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 8c3effbb68eb4433bda2b2597ab1468a from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:56263: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-8c3effbb68eb4433bda2b2597ab1468a
I20240503 02:13:22.824463 12908 raft_consensus.cc:2932] T 004a7ce056044d25ab68a44e51688ca3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.825366 10380 tablet_copy_source_session.cc:205] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.830510 16899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8c3effbb68eb4433bda2b2597ab1468a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.844161 16899 tablet_copy_client.cc:783] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:22.844565 16899 tablet_copy_client.cc:647] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:22.852691 16899 tablet_copy_client.cc:523] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:22.857125 16899 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:22.859969 9943 catalog_manager.cc:5484] T 004a7ce056044d25ab68a44e51688ca3 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 2 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:22.867949 10470 tablet_service.cc:1514] Processing DeleteTablet for tablet 004a7ce056044d25ab68a44e51688ca3 with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:22.879930 16908 tablet_replica.cc:330] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:22.880241 16908 raft_consensus.cc:2229] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:22.880573 16908 raft_consensus.cc:2258] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:22.883203 16899 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:22.883848 16899 tablet_bootstrap.cc:492] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:22.884186 16899 ts_tablet_manager.cc:1367] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.027s user 0.008s sys 0.004s
I20240503 02:13:22.885799 16899 raft_consensus.cc:348] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:22.886188 16899 raft_consensus.cc:729] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:22.886521 16899 consensus_queue.cc:260] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:22.888895 16899 ts_tablet_manager.cc:1398] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20240503 02:13:22.889027 16708 raft_consensus.cc:1050] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 to VOTER
I20240503 02:13:22.889977 16708 consensus_queue.cc:237] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:22.890548 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-8c3effbb68eb4433bda2b2597ab1468a received from {username='slave'} at 127.4.5.69:56263
I20240503 02:13:22.890663 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-8c3effbb68eb4433bda2b2597ab1468a on tablet 8c3effbb68eb4433bda2b2597ab1468a with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:22.891971 16908 ts_tablet_manager.cc:1875] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:22.893543 16873 ts_tablet_manager.cc:902] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:22.903743 13302 raft_consensus.cc:1261] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:22.905292 16280 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:22.906169 12908 raft_consensus.cc:1261] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:22.908274 16707 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:22.909750 16908 ts_tablet_manager.cc:1888] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.3
I20240503 02:13:22.909945 16908 log.cc:1196] T 004a7ce056044d25ab68a44e51688ca3 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/004a7ce056044d25ab68a44e51688ca3
I20240503 02:13:22.910907 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 004a7ce056044d25ab68a44e51688ca3 (table rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]) successfully deleted
I20240503 02:13:22.911406 16859 ts_tablet_manager.cc:902] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:22.912914 13039 raft_consensus.cc:1261] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:22.913633 16280 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:22.914476 16707 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:22.918843 13302 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:22.922907 13039 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:22.933713 16765 raft_consensus.cc:1050] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:22.934765 16765 consensus_queue.cc:237] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:22.947399 9943 catalog_manager.cc:5484] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:22.949364 10489 raft_consensus.cc:1203] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 LEARNER]: Deduplicated request from leader. Original: 2.5->[2.6-2.6] Dedup: 2.6->[]
I20240503 02:13:22.952674 16906 raft_consensus.cc:1050] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:22.954351 12908 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:22.953680 16906 consensus_queue.cc:237] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:22.960492 10089 raft_consensus.cc:1203] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:22.967624 10359 consensus_queue.cc:237] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:22.973171 16859 tablet_copy_client.cc:308] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.71:46557
I20240503 02:13:22.973939 13195 tablet_copy_service.cc:140] P f93b103af083471b8e8c0376e9148aad: Received BeginTabletCopySession request for tablet 918c524462ad41008d2f54f8e954e9bd from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:50947)
I20240503 02:13:22.974162 13195 tablet_copy_service.cc:161] P f93b103af083471b8e8c0376e9148aad: Beginning new tablet copy session on tablet 918c524462ad41008d2f54f8e954e9bd from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:50947: session id = 031a0afb01b6469299fa829fa3467ef7-918c524462ad41008d2f54f8e954e9bd
I20240503 02:13:22.975952 13040 raft_consensus.cc:1261] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:22.976672 16280 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:22.977553 13195 tablet_copy_source_session.cc:205] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:22.980780 16859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 918c524462ad41008d2f54f8e954e9bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:22.981150 13040 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:22.999049 10224 consensus_queue.cc:237] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:22.999434 9943 catalog_manager.cc:5484] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:23.006692 16873 tablet_copy_client.cc:308] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:23.008381 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet 317ce62718cb412bbf04569cab14309f from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:38469)
I20240503 02:13:23.008638 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet 317ce62718cb412bbf04569cab14309f from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:38469: session id = c6ec2f1283994a3089741715f811c850-317ce62718cb412bbf04569cab14309f
I20240503 02:13:23.009972 13168 raft_consensus.cc:1261] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.010695 16297 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.011961 13061 tablet_copy_source_session.cc:205] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:23.012706 16859 tablet_copy_client.cc:783] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:23.013039 16859 tablet_copy_client.cc:647] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
W20240503 02:13:23.015373 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ac9acb6fb9c940ce83922596c849c4d8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:23.016212 10089 raft_consensus.cc:1261] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.016831 16297 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.017802 16859 tablet_copy_client.cc:523] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:23.031019 16730 raft_consensus.cc:1050] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:23.032106 16859 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:23.032284 16730 consensus_queue.cc:237] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:23.037822 16873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 317ce62718cb412bbf04569cab14309f. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.043092 16899 ts_tablet_manager.cc:902] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:23.043723 16707 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:23.052075 16859 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:23.053206 16873 tablet_copy_client.cc:783] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:23.053485 16873 tablet_copy_client.cc:647] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:23.056315 16859 tablet_bootstrap.cc:492] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:23.056644 16859 ts_tablet_manager.cc:1367] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20240503 02:13:23.057407 16859 raft_consensus.cc:348] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:23.057694 16859 raft_consensus.cc:729] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:23.057893 16859 consensus_queue.cc:260] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:23.058490 16859 ts_tablet_manager.cc:1398] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:23.059388 13195 tablet_copy_service.cc:342] P f93b103af083471b8e8c0376e9148aad: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-918c524462ad41008d2f54f8e954e9bd received from {username='slave'} at 127.4.5.65:50947
I20240503 02:13:23.059576 13195 tablet_copy_service.cc:433] P f93b103af083471b8e8c0376e9148aad: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-918c524462ad41008d2f54f8e954e9bd on tablet 918c524462ad41008d2f54f8e954e9bd with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:23.067052 16501 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:23.068951 16873 tablet_copy_client.cc:523] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:23.069774 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet d2d4b31cd7334ed6b18288c1b6d3f958 with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:23.079552 10089 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:23.083675 9950 catalog_manager.cc:5484] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.100908 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d2d4b31cd7334ed6b18288c1b6d3f958 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:23.119324 16720 raft_consensus.cc:1050] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:23.120667 16720 consensus_queue.cc:237] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:23.134946 16937 tablet_replica.cc:330] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:23.135334 16937 raft_consensus.cc:2229] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:23.135792 16937 raft_consensus.cc:2258] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:23.137300 16937 ts_tablet_manager.cc:1875] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:23.142151 16922 ts_tablet_manager.cc:902] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:23.145514 10089 raft_consensus.cc:1261] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.146564 16942 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.148820 10092 raft_consensus.cc:1261] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.149921 16731 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.151120 16899 tablet_copy_client.cc:308] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:23.154634 13303 raft_consensus.cc:1261] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.154904 13039 raft_consensus.cc:1261] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.156098 13039 raft_consensus.cc:1261] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.156491 16731 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.167423 16939 ts_tablet_manager.cc:902] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:23.169001 10227 raft_consensus.cc:1261] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.169977 16671 consensus_queue.cc:1035] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.170111 13166 raft_consensus.cc:1261] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.171872 10358 raft_consensus.cc:1261] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.172086 12907 raft_consensus.cc:1261] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.172881 16214 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.173918 16214 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.174592 16214 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.179680 12906 raft_consensus.cc:1261] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.213217 16939 tablet_copy_client.cc:308] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:23.215456 10491 raft_consensus.cc:1261] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.216531 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet 189d7bee0d6f40a69f25aa578f3d1dec from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:35461)
I20240503 02:13:23.216781 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet 189d7bee0d6f40a69f25aa578f3d1dec from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:35461: session id = c6ec2f1283994a3089741715f811c850-189d7bee0d6f40a69f25aa578f3d1dec
I20240503 02:13:23.214090 13168 raft_consensus.cc:1261] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.218870 15059 consensus_queue.cc:1035] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.222644 10224 raft_consensus.cc:1203] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:23.223367 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 9f8b006a679a4b8d8d3551310b78fb3a from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:59439)
I20240503 02:13:23.223538 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 9f8b006a679a4b8d8d3551310b78fb3a from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:59439: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-9f8b006a679a4b8d8d3551310b78fb3a
I20240503 02:13:23.229465 10246 tablet_copy_source_session.cc:205] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:23.232219 16873 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:23.237784 12928 tablet_copy_source_session.cc:205] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:23.240128 16317 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.021s
I20240503 02:13:23.242957 13172 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:23.254002 16899 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f8b006a679a4b8d8d3551310b78fb3a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.257689 16939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 189d7bee0d6f40a69f25aa578f3d1dec. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.265100 16297 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.267814 12908 raft_consensus.cc:1261] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.270952 16899 tablet_copy_client.cc:783] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:23.271327 16899 tablet_copy_client.cc:647] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:23.271224 16906 consensus_queue.cc:1035] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.287093 16719 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.290683 16754 raft_consensus.cc:1050] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:23.292003 16754 consensus_queue.cc:237] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:23.294634 10359 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.295964 12906 raft_consensus.cc:1203] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:23.297863 9943 catalog_manager.cc:5484] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:23.305630 12906 raft_consensus.cc:1261] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.308957 15059 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.310958 16899 tablet_copy_client.cc:523] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:23.315276 12906 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.319998 10224 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.321692 9943 catalog_manager.cc:5484] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.325042 10359 raft_consensus.cc:1261] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.326092 16899 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:23.338644 13166 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.357081 13168 consensus_queue.cc:237] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:23.361574 12908 raft_consensus.cc:1261] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.362589 16880 consensus_queue.cc:1035] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.362877 10227 raft_consensus.cc:1261] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.364897 16880 consensus_queue.cc:1035] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.366523 16708 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.366703 16297 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.366475 16880 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.373634 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0f6056d9c9a949a6907510bd619825f6 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:23.376391 12908 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.377974 9950 catalog_manager.cc:5484] T 0f6056d9c9a949a6907510bd619825f6 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.378619 16967 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.379523 16967 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.380685 13303 raft_consensus.cc:2932] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } }
I20240503 02:13:23.383739 10224 raft_consensus.cc:2932] T 0f6056d9c9a949a6907510bd619825f6 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
W20240503 02:13:23.389123 12976 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 85427.3us
I20240503 02:13:23.391188 16501 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.392652 16731 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.394379 10470 tablet_service.cc:1514] Processing DeleteTablet for tablet 0f6056d9c9a949a6907510bd619825f6 with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:23.394796 16908 tablet_replica.cc:330] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:23.395047 16908 raft_consensus.cc:2229] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:23.395308 16908 pending_rounds.cc:62] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: Trying to abort 1 pending ops.
I20240503 02:13:23.395418 16908 pending_rounds.cc:68] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023421042495791104 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "0f6056d9c9a949a6907510bd619825f6" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } } }
I20240503 02:13:23.395871 16908 raft_consensus.cc:2894] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 02:13:23.395989 16908 raft_consensus.cc:2866] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:23.396073 16908 raft_consensus.cc:2258] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:23.404991 13039 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.409420 13038 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.413549 10490 raft_consensus.cc:1261] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.418329 13303 consensus_queue.cc:237] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:23.424216 10359 raft_consensus.cc:1261] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:23.424525 10159 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.69:33471, user_credentials={real_user=slave}} blocked reactor thread for 57100.5us
I20240503 02:13:23.425611 16719 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.428988 16754 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20240503 02:13:23.429275 16214 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.434089 16908 ts_tablet_manager.cc:1875] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:23.448199 10359 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.461650 16939 tablet_copy_client.cc:783] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:23.462064 16939 tablet_copy_client.cc:647] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:23.464910 12904 raft_consensus.cc:1261] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.471789 16720 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.473390 12904 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.479049 13168 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.490751 16754 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.491441 16939 tablet_copy_client.cc:523] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:23.486841 12904 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.507846 16922 tablet_copy_client.cc:308] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.72:33671
I20240503 02:13:23.517357 10488 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.520327 10089 raft_consensus.cc:1203] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:23.521798 10092 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.523406 10358 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.526723 16908 ts_tablet_manager.cc:1888] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:23.530126 9952 catalog_manager.cc:5484] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:23.534336 12906 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.538657 12905 consensus_queue.cc:237] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:23.542945 13168 raft_consensus.cc:1261] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.542906 9947 catalog_manager.cc:5484] T c2a328e790e844b5aa9a1cf0fa16ee8c P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.544761 16731 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.546940 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet c2a328e790e844b5aa9a1cf0fa16ee8c with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:23.549201 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet c2a328e790e844b5aa9a1cf0fa16ee8c with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:23.560971 13330 tablet_copy_service.cc:140] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Received BeginTabletCopySession request for tablet 09468c67719244c18d200463892a4f84 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:38343)
I20240503 02:13:23.561223 13330 tablet_copy_service.cc:161] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Beginning new tablet copy session on tablet 09468c67719244c18d200463892a4f84 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:38343: session id = 47f901e617024baaa1865d12261ef84f-09468c67719244c18d200463892a4f84
I20240503 02:13:23.564275 13330 tablet_copy_source_session.cc:205] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:23.568619 16937 ts_tablet_manager.cc:1888] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:23.568902 16937 log.cc:1196] T d2d4b31cd7334ed6b18288c1b6d3f958 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/d2d4b31cd7334ed6b18288c1b6d3f958
I20240503 02:13:23.570072 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet d2d4b31cd7334ed6b18288c1b6d3f958 (table rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]) successfully deleted
I20240503 02:13:23.572136 9950 catalog_manager.cc:5484] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.573869 10091 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.575029 16928 raft_consensus.cc:1050] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:23.576006 16928 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:23.578226 16922 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09468c67719244c18d200463892a4f84. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.578115 9947 catalog_manager.cc:5484] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.582949 16989 tablet_replica.cc:330] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:23.583241 16989 raft_consensus.cc:2229] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:23.584095 16989 raft_consensus.cc:2258] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:23.585187 16989 ts_tablet_manager.cc:1875] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:23.584909 16984 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.586494 16297 raft_consensus.cc:1050] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:23.587316 16297 consensus_queue.cc:237] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:23.590034 16922 tablet_copy_client.cc:783] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:23.590318 16922 tablet_copy_client.cc:647] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:23.592375 16922 tablet_copy_client.cc:523] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:23.592417 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 587a3f3b0f7046d9a840d006c2f8741f with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:23.596140 16908 log.cc:1196] T 0f6056d9c9a949a6907510bd619825f6 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/0f6056d9c9a949a6907510bd619825f6
I20240503 02:13:23.527606 16939 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:23.596626 16922 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:23.597595 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet 0f6056d9c9a949a6907510bd619825f6 (table rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]) successfully deleted
I20240503 02:13:23.599228 12905 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
W20240503 02:13:23.603381 12975 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 46362.7us
I20240503 02:13:23.605031 13040 raft_consensus.cc:1261] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.606045 16730 consensus_queue.cc:1035] T 587a3f3b0f7046d9a840d006c2f8741f P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.608300 13039 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.611199 12904 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:23.611428 10491 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:23.613463 13166 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:23.614042 16708 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:23.615666 16708 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:23.618455 13168 raft_consensus.cc:2932] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.585945 16873 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:23.622813 16873 tablet_bootstrap.cc:492] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:23.623186 16873 ts_tablet_manager.cc:1367] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.391s user 0.010s sys 0.000s
I20240503 02:13:23.624125 16873 raft_consensus.cc:348] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:23.624429 16873 raft_consensus.cc:729] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:23.624712 16873 consensus_queue.cc:260] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:23.625434 16873 ts_tablet_manager.cc:1398] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20240503 02:13:23.625712 10092 raft_consensus.cc:1261] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.626403 16765 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.627717 16708 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:23.634896 10227 consensus_queue.cc:237] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:23.645042 16939 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:23.645763 16939 tablet_bootstrap.cc:492] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:23.646221 16939 ts_tablet_manager.cc:1367] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.119s user 0.007s sys 0.004s
I20240503 02:13:23.647178 16939 raft_consensus.cc:348] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:23.648070 16939 raft_consensus.cc:729] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:23.649077 16939 consensus_queue.cc:260] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:23.653820 10092 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:23.653860 10162 consensus_peers.cc:485] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 13e83ce8820b491992c041a8c21fa3df. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:23.654745 16297 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:23.665169 16939 ts_tablet_manager.cc:1398] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20240503 02:13:23.665756 16731 raft_consensus.cc:1050] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:23.666712 16731 consensus_queue.cc:237] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:23.668908 9951 catalog_manager.cc:5484] T 587a3f3b0f7046d9a840d006c2f8741f P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.670663 16922 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:23.671787 16922 tablet_bootstrap.cc:492] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:23.627636 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-317ce62718cb412bbf04569cab14309f received from {username='slave'} at 127.4.5.68:38469
I20240503 02:13:23.672252 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session c6ec2f1283994a3089741715f811c850-317ce62718cb412bbf04569cab14309f on tablet 317ce62718cb412bbf04569cab14309f with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:23.672086 16922 ts_tablet_manager.cc:1367] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.004s
I20240503 02:13:23.675550 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-189d7bee0d6f40a69f25aa578f3d1dec received from {username='slave'} at 127.4.5.68:35461
I20240503 02:13:23.675753 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session c6ec2f1283994a3089741715f811c850-189d7bee0d6f40a69f25aa578f3d1dec on tablet 189d7bee0d6f40a69f25aa578f3d1dec with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:23.683655 10227 raft_consensus.cc:1261] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.686081 13038 raft_consensus.cc:1261] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.686158 16996 ts_tablet_manager.cc:902] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:23.689754 16922 raft_consensus.cc:348] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:23.690116 16922 raft_consensus.cc:729] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:23.690335 16922 consensus_queue.cc:260] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:23.690438 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 587a3f3b0f7046d9a840d006c2f8741f with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:23.691223 16922 ts_tablet_manager.cc:1398] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20240503 02:13:23.695935 13303 raft_consensus.cc:1261] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.697212 13330 tablet_copy_service.cc:342] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-09468c67719244c18d200463892a4f84 received from {username='slave'} at 127.4.5.67:38343
I20240503 02:13:23.697355 13330 tablet_copy_service.cc:433] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: ending tablet copy session 47f901e617024baaa1865d12261ef84f-09468c67719244c18d200463892a4f84 on tablet 09468c67719244c18d200463892a4f84 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:23.699518 16730 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.015s
I20240503 02:13:23.705168 12907 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:23.705920 16984 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.707988 13303 raft_consensus.cc:1261] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.711089 12905 raft_consensus.cc:1261] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.714489 16731 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.718154 10488 raft_consensus.cc:1203] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:23.730383 16297 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:23.737514 16707 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 6 to 7, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:23.740578 10224 consensus_queue.cc:237] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:23.745165 17005 tablet_replica.cc:330] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:23.745086 16730 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.745417 17005 raft_consensus.cc:2229] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:23.746104 17005 raft_consensus.cc:2258] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:23.747761 13302 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.755992 16708 raft_consensus.cc:1050] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:23.756968 16708 consensus_queue.cc:237] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:23.755407 13038 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.759886 13040 raft_consensus.cc:1261] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.760802 16501 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.760433 12906 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:23.765568 9951 catalog_manager.cc:5484] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.773195 16989 ts_tablet_manager.cc:1888] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:23.773491 16989 log.cc:1196] T c2a328e790e844b5aa9a1cf0fa16ee8c P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/c2a328e790e844b5aa9a1cf0fa16ee8c
I20240503 02:13:23.774453 16945 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.774348 10089 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:23.778085 17005 ts_tablet_manager.cc:1875] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:23.774747 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet c2a328e790e844b5aa9a1cf0fa16ee8c (table rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]) successfully deleted
I20240503 02:13:23.779048 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ccd3faa7afbb4b92a4a4c4645e477771 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:23.780853 16501 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.785643 10227 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.786782 16501 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:23.787150 9950 catalog_manager.cc:5484] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:23.787647 16501 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.783049 13040 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.797749 13172 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:23.805214 12904 raft_consensus.cc:1261] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.809716 10224 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.810559 16280 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.812002 10359 raft_consensus.cc:1203] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:23.813383 12904 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.821522 16996 tablet_copy_client.cc:308] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:23.822806 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet ac9acb6fb9c940ce83922596c849c4d8 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:35599)
I20240503 02:13:23.823000 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet ac9acb6fb9c940ce83922596c849c4d8 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:35599: session id = 47f901e617024baaa1865d12261ef84f-ac9acb6fb9c940ce83922596c849c4d8
I20240503 02:13:23.826692 10246 tablet_copy_source_session.cc:205] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:23.834234 16996 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ac9acb6fb9c940ce83922596c849c4d8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:23.841289 10488 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 6 to 7, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:23.845541 16297 consensus_queue.cc:1035] T ccd3faa7afbb4b92a4a4c4645e477771 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.852104 10491 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:23.852854 16707 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.854313 16707 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:23.870842 13040 consensus_queue.cc:237] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:23.885016 9950 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 6 to 7, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.885911 13038 consensus_queue.cc:237] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:23.886936 12904 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:23.888741 10359 raft_consensus.cc:1261] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.889436 16316 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.892565 16754 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.895282 10490 raft_consensus.cc:1203] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:13:23.896423 10488 raft_consensus.cc:1261] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.897152 16316 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.897989 9952 catalog_manager.cc:5484] T ccd3faa7afbb4b92a4a4c4645e477771 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:23.898000 10359 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.899057 10488 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.913872 12905 consensus_queue.cc:237] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:23.916074 13303 raft_consensus.cc:1261] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.916256 10224 raft_consensus.cc:1261] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:23.916615 16996 tablet_copy_client.cc:783] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:23.916930 16996 tablet_copy_client.cc:647] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:23.918416 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09e0373a72654a1082464fac8d428483 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:23.923171 16996 tablet_copy_client.cc:523] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:23.924050 13172 raft_consensus.cc:1261] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.924793 9943 catalog_manager.cc:5484] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:23.925328 16977 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.926892 16996 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:23.928830 16317 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.012s
W20240503 02:13:23.931509 12975 consensus_peers.cc:485] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e9def231ec4a43349b5786776e6c57b7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:23.932044 16317 consensus_queue.cc:1035] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:23.933143 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet ccd3faa7afbb4b92a4a4c4645e477771 with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:23.936018 16969 raft_consensus.cc:1050] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:23.936954 16969 consensus_queue.cc:237] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:23.937708 12905 raft_consensus.cc:1261] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:23.938688 16977 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:23.940091 10227 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:23.950525 10359 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 2.7, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:23.951833 16297 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:23.957985 10091 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:23.961817 13303 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:23.972569 10488 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:23.974856 17005 ts_tablet_manager.cc:1888] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:23.975039 17005 log.cc:1196] T 587a3f3b0f7046d9a840d006c2f8741f P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/587a3f3b0f7046d9a840d006c2f8741f
I20240503 02:13:23.975915 17005 tablet_replica.cc:330] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:23.976178 17005 raft_consensus.cc:2229] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:23.976265 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 587a3f3b0f7046d9a840d006c2f8741f (table rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]) successfully deleted
I20240503 02:13:23.976540 16919 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:23.977614 16977 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:23.978256 13172 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:23.985599 10224 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:23.989769 9952 catalog_manager.cc:5484] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:23.994077 16707 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:23.998019 9950 catalog_manager.cc:5484] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:23.976401 17005 raft_consensus.cc:2258] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.003733 17005 ts_tablet_manager.cc:1875] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.004066 10488 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:24.009850 16317 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:24.015390 16707 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [term 2 LEADER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.020133 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet db2da39193164d1ebf830282258d47d7 with cas_config_opid_index 7: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.021092 10490 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.027040 16996 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:24.027508 16996 tablet_bootstrap.cc:492] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:24.027793 16996 ts_tablet_manager.cc:1367] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.101s user 0.002s sys 0.005s
I20240503 02:13:24.029788 9952 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 7 to 8, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 2 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.037551 9951 catalog_manager.cc:5484] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:24.042634 10091 raft_consensus.cc:1261] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.048125 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 09e0373a72654a1082464fac8d428483 with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:24.049850 9952 catalog_manager.cc:5484] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:24.054386 16937 tablet_replica.cc:330] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:24.054795 16937 raft_consensus.cc:2229] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.055706 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.058393 16937 raft_consensus.cc:2258] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.046085 16996 raft_consensus.cc:348] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:24.058790 16996 raft_consensus.cc:729] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:24.059036 16996 consensus_queue.cc:260] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:24.061254 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:24.061856 10227 consensus_queue.cc:237] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:24.065100 10091 raft_consensus.cc:1261] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.065704 16297 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.068965 16501 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.073786 12907 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 2 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:24.080479 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet db2da39193164d1ebf830282258d47d7 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 8) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:24.080965 16989 tablet_replica.cc:330] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:24.081250 16989 raft_consensus.cc:2229] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.081518 16989 raft_consensus.cc:2258] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.082870 16989 ts_tablet_manager.cc:1875] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.082929 16919 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:24.083346 12905 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.089378 16937 ts_tablet_manager.cc:1875] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.091599 16996 ts_tablet_manager.cc:1398] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.064s user 0.004s sys 0.000s
I20240503 02:13:24.092978 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-ac9acb6fb9c940ce83922596c849c4d8 received from {username='slave'} at 127.4.5.67:35599
I20240503 02:13:24.093132 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 47f901e617024baaa1865d12261ef84f-ac9acb6fb9c940ce83922596c849c4d8 on tablet ac9acb6fb9c940ce83922596c849c4d8 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:24.095671 13302 raft_consensus.cc:2932] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:24.098312 16989 ts_tablet_manager.cc:1888] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.7
I20240503 02:13:24.098475 16989 log.cc:1196] T db2da39193164d1ebf830282258d47d7 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/db2da39193164d1ebf830282258d47d7
I20240503 02:13:24.102588 17005 ts_tablet_manager.cc:1888] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:24.104127 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet db2da39193164d1ebf830282258d47d7 (table rebalancer_extra_table_14 [id=e84150c537a841ed8d38c44f2e31f512]) successfully deleted
I20240503 02:13:24.105033 12907 raft_consensus.cc:2932] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.8: config changed from index 7 to 8, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.109297 17005 log.cc:1196] T ccd3faa7afbb4b92a4a4c4645e477771 P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/ccd3faa7afbb4b92a4a4c4645e477771
W20240503 02:13:24.111548 13107 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 110947us
I20240503 02:13:24.114358 13303 raft_consensus.cc:1261] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.115315 12904 raft_consensus.cc:1261] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.120361 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet ccd3faa7afbb4b92a4a4c4645e477771 (table rebalancer_extra_table_40 [id=a063a9a6a9a0414b8ded1f32843d825c]) successfully deleted
I20240503 02:13:24.128065 12903 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:24.149734 16317 raft_consensus.cc:1050] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:24.151158 16317 consensus_queue.cc:237] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:24.154639 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0df3294fa2d74baea8c573a5dcfa96bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.160712 12902 raft_consensus.cc:1261] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.161521 10091 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.169126 16937 ts_tablet_manager.cc:1888] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:24.169389 16937 log.cc:1196] T 09e0373a72654a1082464fac8d428483 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/09e0373a72654a1082464fac8d428483
I20240503 02:13:24.171136 12906 raft_consensus.cc:2932] T ccd3faa7afbb4b92a4a4c4645e477771 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:24.173110 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 09e0373a72654a1082464fac8d428483 (table rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]) successfully deleted
I20240503 02:13:24.045190 17032 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.132s
I20240503 02:13:24.176051 12902 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:24.177121 17033 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.177922 16906 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.191416 9951 catalog_manager.cc:5484] T 0df3294fa2d74baea8c573a5dcfa96bd P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.197479 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 0df3294fa2d74baea8c573a5dcfa96bd with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:24.199746 17036 tablet_replica.cc:330] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:24.200019 17036 raft_consensus.cc:2229] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.207549 16501 consensus_queue.cc:1035] T 0df3294fa2d74baea8c573a5dcfa96bd P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.200313 17036 raft_consensus.cc:2258] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.211614 17036 ts_tablet_manager.cc:1875] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.217098 10353 raft_consensus.cc:1203] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:24.224611 16982 raft_consensus.cc:1050] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:24.225672 16982 consensus_queue.cc:237] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:24.229931 12908 raft_consensus.cc:2932] T 0df3294fa2d74baea8c573a5dcfa96bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.228029 17047 ts_tablet_manager.cc:902] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:24.287117 10353 raft_consensus.cc:1261] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.288411 13040 raft_consensus.cc:1261] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.288620 12908 raft_consensus.cc:1261] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.288829 17042 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.289324 16214 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.289968 16214 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.297350 10227 raft_consensus.cc:1261] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.297996 16984 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.299077 17036 ts_tablet_manager.cc:1888] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:24.299273 17036 log.cc:1196] T 90b2d976f9fe42d2882b1c1ef7a01eb3 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/90b2d976f9fe42d2882b1c1ef7a01eb3
I20240503 02:13:24.307547 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 90b2d976f9fe42d2882b1c1ef7a01eb3 (table rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]) successfully deleted
I20240503 02:13:24.309739 10490 raft_consensus.cc:1261] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.310634 16974 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.333030 13295 raft_consensus.cc:1261] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.347702 16974 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.359115 17042 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.362597 17032 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.369145 12908 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.370203 12903 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.373395 10353 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.380492 10091 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.382807 9951 catalog_manager.cc:5484] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.401773 13302 consensus_queue.cc:237] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:24.403107 9950 catalog_manager.cc:5484] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:24.404911 10359 raft_consensus.cc:1261] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.405665 17042 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.410919 16974 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.412060 13172 consensus_queue.cc:237] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:24.413983 12903 raft_consensus.cc:1261] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.415427 16982 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.418217 12903 raft_consensus.cc:1261] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.417387 10091 raft_consensus.cc:1261] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.420112 17033 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.420636 17033 consensus_queue.cc:1035] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.420547 13302 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.423903 17033 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.425081 10089 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.433028 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 918c524462ad41008d2f54f8e954e9bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.437587 12906 raft_consensus.cc:2932] T 918c524462ad41008d2f54f8e954e9bd P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:24.437970 10227 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.439689 17042 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.445436 10359 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.447237 9947 catalog_manager.cc:5484] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
W20240503 02:13:24.449719 9939 catalog_manager.cc:5007] ChangeConfig:REMOVE_PEER RPC for tablet 918c524462ad41008d2f54f8e954e9bd with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER failed with leader f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557) due to CAS failure; no further retry: Illegal state: Request specified cas_config_opid_index of 3 but the committed config has opid_index of 4
I20240503 02:13:24.453816 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 09468c67719244c18d200463892a4f84 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.457156 16899 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:24.457237 16316 raft_consensus.cc:1050] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:24.457759 10488 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.458243 16316 consensus_queue.cc:237] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:24.465863 16899 tablet_bootstrap.cc:492] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:24.466256 17046 tablet_replica.cc:330] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:24.466260 16899 ts_tablet_manager.cc:1367] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 1.140s user 0.058s sys 0.002s
I20240503 02:13:24.467000 17046 raft_consensus.cc:2229] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.467432 17046 raft_consensus.cc:2258] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.467352 16899 raft_consensus.cc:348] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:24.467681 16899 raft_consensus.cc:729] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:24.467919 16899 consensus_queue.cc:260] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:24.468705 17046 ts_tablet_manager.cc:1875] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.471486 9947 catalog_manager.cc:5484] T 09468c67719244c18d200463892a4f84 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.478387 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet 09468c67719244c18d200463892a4f84 with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:24.480542 17036 tablet_replica.cc:330] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:24.480856 17036 raft_consensus.cc:2229] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.481071 17036 pending_rounds.cc:62] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: Trying to abort 1 pending ops.
I20240503 02:13:24.481387 12906 raft_consensus.cc:1203] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:24.481201 17036 pending_rounds.cc:68] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023421047705686016 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "09468c67719244c18d200463892a4f84" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } } }
I20240503 02:13:24.481292 9951 catalog_manager.cc:5484] T 918c524462ad41008d2f54f8e954e9bd P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:24.481748 17036 raft_consensus.cc:2894] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 02:13:24.481874 17036 raft_consensus.cc:2866] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:24.481997 17036 raft_consensus.cc:2258] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.484375 17036 ts_tablet_manager.cc:1875] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.484982 16899 ts_tablet_manager.cc:1398] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20240503 02:13:24.485781 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-9f8b006a679a4b8d8d3551310b78fb3a received from {username='slave'} at 127.4.5.69:59439
I20240503 02:13:24.485917 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-9f8b006a679a4b8d8d3551310b78fb3a on tablet 9f8b006a679a4b8d8d3551310b78fb3a with peer e3cbe34992dc4b77b9aa9e9a4c143e78
W20240503 02:13:24.486227 10026 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 149655us
W20240503 02:13:24.487669 10424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 149918us
I20240503 02:13:24.496256 12906 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.515192 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 918c524462ad41008d2f54f8e954e9bd with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:24.518641 13172 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.518606 10488 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.519213 10353 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:24.519294 16968 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:24.519964 16968 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:24.532052 17068 ts_tablet_manager.cc:902] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:24.532296 17036 ts_tablet_manager.cc:1888] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:24.532810 17036 log.cc:1196] T 09468c67719244c18d200463892a4f84 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/09468c67719244c18d200463892a4f84
I20240503 02:13:24.533694 16977 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.007s
W20240503 02:13:24.571286 12975 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 51426us
I20240503 02:13:24.584013 16977 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.585788 17068 tablet_copy_client.cc:308] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.70:43707
I20240503 02:13:24.586875 12906 consensus_queue.cc:237] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:24.572804 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 09468c67719244c18d200463892a4f84 (table rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]) successfully deleted
I20240503 02:13:24.589545 10227 raft_consensus.cc:1261] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.589406 13168 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.589952 10488 raft_consensus.cc:1261] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.590936 13061 tablet_copy_service.cc:140] P 8d454a3607d042dc9559ed59c3ccd84b: Received BeginTabletCopySession request for tablet e9def231ec4a43349b5786776e6c57b7 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:38795)
I20240503 02:13:24.591223 13061 tablet_copy_service.cc:161] P 8d454a3607d042dc9559ed59c3ccd84b: Beginning new tablet copy session on tablet e9def231ec4a43349b5786776e6c57b7 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:38795: session id = eff8e8d11a4447218741bb994aa41082-e9def231ec4a43349b5786776e6c57b7
I20240503 02:13:24.591464 10488 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.592865 16984 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.596746 13061 tablet_copy_source_session.cc:205] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:24.600117 17068 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e9def231ec4a43349b5786776e6c57b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.601518 9951 catalog_manager.cc:5484] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.604905 17047 tablet_copy_client.cc:308] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:24.610649 17046 ts_tablet_manager.cc:1888] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:24.610858 17046 log.cc:1196] T 0df3294fa2d74baea8c573a5dcfa96bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/0df3294fa2d74baea8c573a5dcfa96bd
I20240503 02:13:24.614317 17046 tablet_replica.cc:330] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:24.614689 17046 raft_consensus.cc:2229] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.615001 17046 pending_rounds.cc:62] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Trying to abort 1 pending ops.
I20240503 02:13:24.615200 17046 pending_rounds.cc:68] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023421046523133952 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "918c524462ad41008d2f54f8e954e9bd" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } } }
I20240503 02:13:24.615823 17046 raft_consensus.cc:2894] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 02:13:24.616010 17046 raft_consensus.cc:2866] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:24.616210 17046 raft_consensus.cc:2258] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.617480 17046 ts_tablet_manager.cc:1875] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.619767 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 0df3294fa2d74baea8c573a5dcfa96bd (table rebalancer_extra_table_30 [id=68d27f1026654689b74172738a901503]) successfully deleted
I20240503 02:13:24.621886 16984 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.627328 17046 ts_tablet_manager.cc:1888] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:24.627626 17046 log.cc:1196] T 918c524462ad41008d2f54f8e954e9bd P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/918c524462ad41008d2f54f8e954e9bd
I20240503 02:13:24.628645 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 189d7bee0d6f40a69f25aa578f3d1dec with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.629177 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 918c524462ad41008d2f54f8e954e9bd (table rebalancer_extra_table_0 [id=8643bb84a04c4753a57c38525ecbdc5b]) successfully deleted
I20240503 02:13:24.630594 17068 tablet_copy_client.cc:783] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:24.630851 17068 tablet_copy_client.cc:647] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:24.631767 10490 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.632726 16297 raft_consensus.cc:1050] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:24.633597 16297 consensus_queue.cc:237] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:24.638127 17068 tablet_copy_client.cc:523] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:24.641573 9951 catalog_manager.cc:5484] T 189d7bee0d6f40a69f25aa578f3d1dec P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.643361 16984 consensus_queue.cc:1035] T 189d7bee0d6f40a69f25aa578f3d1dec P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.645960 13168 raft_consensus.cc:1261] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.647527 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 189d7bee0d6f40a69f25aa578f3d1dec with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:24.647814 16501 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.648068 17046 tablet_replica.cc:330] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:24.648313 17046 raft_consensus.cc:2229] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.648661 17046 raft_consensus.cc:2258] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.649020 17068 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:24.649516 10353 raft_consensus.cc:1261] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.649791 17046 ts_tablet_manager.cc:1875] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.651540 16501 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.655674 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 13e83ce8820b491992c041a8c21fa3df from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:38111)
I20240503 02:13:24.655905 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 13e83ce8820b491992c041a8c21fa3df from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:38111: session id = 8d454a3607d042dc9559ed59c3ccd84b-13e83ce8820b491992c041a8c21fa3df
I20240503 02:13:24.657624 17046 ts_tablet_manager.cc:1888] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:24.658547 17046 log.cc:1196] T 189d7bee0d6f40a69f25aa578f3d1dec P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/189d7bee0d6f40a69f25aa578f3d1dec
I20240503 02:13:24.659649 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 189d7bee0d6f40a69f25aa578f3d1dec (table rebalancer_extra_table_34 [id=ad3e3abfd1174d56aeb54a424aff2f4d]) successfully deleted
I20240503 02:13:24.659379 10359 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.659844 10246 tablet_copy_source_session.cc:205] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:24.664597 10091 raft_consensus.cc:1261] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:24.673576 17047 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 13e83ce8820b491992c041a8c21fa3df. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:24.679140 16501 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:24.680956 17068 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:24.681416 17068 tablet_bootstrap.cc:492] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:24.681715 17068 ts_tablet_manager.cc:1367] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.033s user 0.012s sys 0.000s
I20240503 02:13:24.682219 17047 tablet_copy_client.cc:783] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:24.682534 17047 tablet_copy_client.cc:647] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:24.682451 17068 raft_consensus.cc:348] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:24.682703 17068 raft_consensus.cc:729] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:24.682909 17068 consensus_queue.cc:260] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:24.691946 17068 ts_tablet_manager.cc:1398] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20240503 02:13:24.696128 17047 tablet_copy_client.cc:523] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:24.696238 13040 consensus_queue.cc:237] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:24.698774 10491 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:24.699638 16316 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:24.700573 17047 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:24.702850 17069 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.703455 10491 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.708940 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 317ce62718cb412bbf04569cab14309f with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.709692 10359 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:24.712848 10227 raft_consensus.cc:2932] T 189d7bee0d6f40a69f25aa578f3d1dec P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.713402 16501 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.717932 9947 catalog_manager.cc:5484] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 6 to 7, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:24.719779 10359 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.727576 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet 317ce62718cb412bbf04569cab14309f with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:24.726159 13172 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.735263 10089 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.738307 13061 tablet_copy_service.cc:342] P 8d454a3607d042dc9559ed59c3ccd84b: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-e9def231ec4a43349b5786776e6c57b7 received from {username='slave'} at 127.4.5.66:38795
I20240503 02:13:24.738670 13061 tablet_copy_service.cc:433] P 8d454a3607d042dc9559ed59c3ccd84b: ending tablet copy session eff8e8d11a4447218741bb994aa41082-e9def231ec4a43349b5786776e6c57b7 on tablet e9def231ec4a43349b5786776e6c57b7 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:24.738116 9951 catalog_manager.cc:5484] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:24.739219 16977 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.011s
I20240503 02:13:24.745082 10353 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:24.771685 10227 consensus_queue.cc:237] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:24.773852 10353 raft_consensus.cc:1261] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.774475 17087 tablet_replica.cc:330] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:24.774751 10089 raft_consensus.cc:1261] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:24.775417 16501 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.775983 16501 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:24.779578 16941 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.783959 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ac9acb6fb9c940ce83922596c849c4d8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:24.784685 17087 raft_consensus.cc:2229] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.784958 17087 raft_consensus.cc:2258] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.784751 10089 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.787452 17087 ts_tablet_manager.cc:1875] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.793499 10353 raft_consensus.cc:2932] T ac9acb6fb9c940ce83922596c849c4d8 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.799646 9951 catalog_manager.cc:5484] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:24.806361 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet ac9acb6fb9c940ce83922596c849c4d8 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:24.816501 17087 ts_tablet_manager.cc:1888] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:24.823253 17087 log.cc:1196] T 317ce62718cb412bbf04569cab14309f P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/317ce62718cb412bbf04569cab14309f
I20240503 02:13:24.824556 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 317ce62718cb412bbf04569cab14309f (table rebalancer_extra_table_6 [id=d0fb764b8547438abe9baf085416c638]) successfully deleted
I20240503 02:13:24.825230 17087 tablet_replica.cc:330] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:24.825565 17087 raft_consensus.cc:2229] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:24.825923 17087 raft_consensus.cc:2258] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:24.828083 17087 ts_tablet_manager.cc:1875] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:24.842880 17087 ts_tablet_manager.cc:1888] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:24.843063 17087 log.cc:1196] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/ac9acb6fb9c940ce83922596c849c4d8
I20240503 02:13:24.844290 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet ac9acb6fb9c940ce83922596c849c4d8 (table rebalancer_extra_table_44 [id=2cbc9f76eaf346f3a1307398478f34c4]) successfully deleted
I20240503 02:13:24.866786 16297 raft_consensus.cc:1050] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:24.868119 16297 consensus_queue.cc:237] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:24.875116 12908 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:24.876263 10491 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:24.877916 16765 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:24.878497 16765 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:24.881764 13038 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:24.888413 16941 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:24.891202 17047 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:24.891741 17047 tablet_bootstrap.cc:492] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:24.892050 17047 ts_tablet_manager.cc:1367] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.192s user 0.052s sys 0.006s
I20240503 02:13:24.893005 17047 raft_consensus.cc:348] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:24.893329 17047 raft_consensus.cc:729] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:24.893518 17047 consensus_queue.cc:260] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:24.894121 17047 ts_tablet_manager.cc:1398] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20240503 02:13:24.895690 16765 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:24.898330 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-13e83ce8820b491992c041a8c21fa3df received from {username='slave'} at 127.4.5.70:38111
I20240503 02:13:24.898471 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-13e83ce8820b491992c041a8c21fa3df on tablet 13e83ce8820b491992c041a8c21fa3df with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:24.905397 12908 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:24.915292 13040 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:24.918488 9951 catalog_manager.cc:5484] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:24.940613 10491 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:25.007458 10227 raft_consensus.cc:1203] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:25.060572 10353 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a"
dest_uuid: "47f901e617024baaa1865d12261ef84f"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:40328
I20240503 02:13:25.060808 10353 raft_consensus.cc:595] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:25.127732 13040 raft_consensus.cc:1203] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:25.221760 10089 consensus_queue.cc:237] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:25.225212 13305 raft_consensus.cc:1261] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.225589 13172 raft_consensus.cc:1261] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.225857 16653 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:25.226752 10026 consensus_peers.cc:485] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ce47128b75a645e8a9f6e4cbc9d8d021. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.226928 16653 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.237998 17058 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:25.238723 13305 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:25.239635 13172 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:25.245616 9952 catalog_manager.cc:5484] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.261525 16707 raft_consensus.cc:979] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: : Instructing follower e3cbe34992dc4b77b9aa9e9a4c143e78 to start an election
I20240503 02:13:25.261735 16707 raft_consensus.cc:1067] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Signalling peer e3cbe34992dc4b77b9aa9e9a4c143e78 to start an election
I20240503 02:13:25.262562 12908 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a"
dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
from {username='slave'} at 127.4.5.67:52779
I20240503 02:13:25.262845 12908 raft_consensus.cc:482] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:25.262988 12908 raft_consensus.cc:3037] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:25.265663 12908 raft_consensus.cc:504] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:25.267015 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:25.267277 10491 raft_consensus.cc:3037] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:25.267282 12908 leader_election.cc:288] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 2 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:25.269861 10491 raft_consensus.cc:2454] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 2.
I20240503 02:13:25.270992 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:25.271246 10227 raft_consensus.cc:3037] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:25.276331 10353 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8c3effbb68eb4433bda2b2597ab1468a" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "47f901e617024baaa1865d12261ef84f"
I20240503 02:13:25.276613 10353 raft_consensus.cc:3032] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:25.276700 10353 raft_consensus.cc:729] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Running, Role: LEADER
I20240503 02:13:25.276997 10353 consensus_queue.cc:260] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:25.277559 10353 raft_consensus.cc:3037] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:25.291764 10227 raft_consensus.cc:2454] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 2.
I20240503 02:13:25.295642 12842 leader_election.cc:302] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:25.296092 16730 raft_consensus.cc:2781] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:25.296252 16730 raft_consensus.cc:686] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 LEADER]: Becoming Leader. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Running, Role: LEADER
I20240503 02:13:25.296679 16730 consensus_queue.cc:237] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:25.303560 10353 raft_consensus.cc:2454] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e3cbe34992dc4b77b9aa9e9a4c143e78 in term 2.
I20240503 02:13:25.317628 9947 catalog_manager.cc:5484] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69). New cstate: current_term: 2 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:25.324165 10224 consensus_queue.cc:237] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:25.326969 13305 raft_consensus.cc:1261] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:25.327049 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c014b2069ae42788aae87d8304fa155. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.327208 10089 raft_consensus.cc:1261] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.327841 16297 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.328712 16297 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.331243 16941 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:25.331832 10089 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:25.332656 13305 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:25.340507 9947 catalog_manager.cc:5484] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:25.365280 12908 consensus_queue.cc:237] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:25.367889 10353 raft_consensus.cc:1261] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:25.368064 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 254af82e74a045f8a9c9d9de48cdee3e. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.368587 13040 raft_consensus.cc:1261] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.369197 16730 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.369552 16317 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.374202 16730 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:25.377348 10353 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:25.381323 9952 catalog_manager.cc:5484] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.391448 13040 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:25.416265 10089 consensus_queue.cc:237] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:25.420662 10491 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:25.420879 13040 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:25.421097 10029 consensus_peers.cc:485] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc0a049eeb0047558d3c9eb370467c53. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.421648 16653 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:25.422194 16653 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:25.438616 16653 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:25.439565 13040 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:25.446297 9952 catalog_manager.cc:5484] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:25.451220 10490 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:25.488142 12908 consensus_queue.cc:237] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
W20240503 02:13:25.491241 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9fbc1a14c000465e9bb642fcc1e84611. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.493228 16977 raft_consensus.cc:1050] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:25.493283 10359 raft_consensus.cc:1261] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.494217 16977 consensus_queue.cc:237] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:25.494633 13305 raft_consensus.cc:1261] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.495030 16317 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.495841 16984 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.498009 10224 raft_consensus.cc:1261] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:25.498091 12908 raft_consensus.cc:1261] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:25.498076 13168 raft_consensus.cc:1261] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:25.498904 17069 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:25.499681 17069 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:25.500332 17069 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:25.521303 17069 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:25.523667 16730 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:25.524616 13302 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:25.525642 10353 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:25.529119 10224 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:25.531366 9951 catalog_manager.cc:5484] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.531589 9952 catalog_manager.cc:5484] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:25.537458 12906 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:25.554450 13040 consensus_queue.cc:237] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:25.565613 10227 raft_consensus.cc:1261] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:25.565690 13172 raft_consensus.cc:1261] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:25.566558 17069 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:25.568883 13172 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:25.572293 16316 consensus_queue.cc:1035] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.001s
I20240503 02:13:25.578315 16316 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:25.582187 10227 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:25.589345 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e9def231ec4a43349b5786776e6c57b7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:25.590013 9951 catalog_manager.cc:5484] T e9def231ec4a43349b5786776e6c57b7 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:25.594498 13168 raft_consensus.cc:2932] T e9def231ec4a43349b5786776e6c57b7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:25.595317 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet e9def231ec4a43349b5786776e6c57b7 with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:25.605986 17239 tablet_replica.cc:330] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:25.606388 17239 raft_consensus.cc:2229] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:25.606716 17239 raft_consensus.cc:2258] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:25.616667 17239 ts_tablet_manager.cc:1875] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:13:25.623268 10424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 81810.8us
I20240503 02:13:25.633240 17239 ts_tablet_manager.cc:1888] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:25.633738 17239 log.cc:1196] T e9def231ec4a43349b5786776e6c57b7 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/e9def231ec4a43349b5786776e6c57b7
I20240503 02:13:25.635210 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet e9def231ec4a43349b5786776e6c57b7 (table rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]) successfully deleted
I20240503 02:13:25.644644 10227 consensus_queue.cc:237] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:25.647222 12906 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:25.647341 10359 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:25.647846 16765 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:25.648190 16941 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:13:25.648689 10162 consensus_peers.cc:485] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af7beeafa08f477796e5070fbc177dcc. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.651311 16941 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:25.652244 10359 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:25.659142 9943 catalog_manager.cc:5484] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.670289 12906 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:25.682300 10359 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:25.683092 16730 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:25.694329 10490 consensus_queue.cc:237] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
W20240503 02:13:25.698858 10426 consensus_peers.cc:485] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f385792ed7c8465eb7cd64142778c9bf. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.700147 13305 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.700947 13040 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.700919 17236 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.701520 17236 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.705868 17242 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:25.706779 13040 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:25.712754 13305 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:25.715471 9943 catalog_manager.cc:5484] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.755786 16501 raft_consensus.cc:1050] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:25.756294 10091 consensus_queue.cc:237] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:25.757114 16501 consensus_queue.cc:237] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:25.771152 13040 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:25.771816 12906 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:25.771831 16941 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:25.772287 10089 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:25.772428 16765 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:25.775075 16765 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
W20240503 02:13:25.799306 10028 consensus_peers.cc:485] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e4caac5d5607431ba8af43993e9de075. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:25.799928 13040 raft_consensus.cc:1261] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.800184 13172 raft_consensus.cc:1261] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:25.800735 16941 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:25.802011 12908 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:25.802845 16654 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.803544 16654 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:25.806190 13038 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:25.809381 9951 catalog_manager.cc:5484] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:25.812836 10491 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:25.813643 16730 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:25.814715 17222 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:25.820655 10227 consensus_queue.cc:237] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:25.823673 13040 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:25.823499 13038 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:25.824386 16941 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:25.827817 16765 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:25.838409 9952 catalog_manager.cc:5484] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:25.839923 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 13e83ce8820b491992c041a8c21fa3df with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:25.841269 10227 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:25.842885 16984 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:25.847540 13037 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:25.854420 9943 catalog_manager.cc:5484] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 8 to 9, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:25.863710 10072 tablet_service.cc:1514] Processing DeleteTablet for tablet 13e83ce8820b491992c041a8c21fa3df with delete_type TABLET_DATA_TOMBSTONED (TS 031a0afb01b6469299fa829fa3467ef7 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:36690
I20240503 02:13:25.859957 10091 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:25.881172 12908 raft_consensus.cc:1261] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:25.881862 16941 consensus_queue.cc:1035] T 13e83ce8820b491992c041a8c21fa3df P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:25.884093 12906 raft_consensus.cc:2932] T 13e83ce8820b491992c041a8c21fa3df P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:25.887524 17274 ts_tablet_manager.cc:902] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:25.893791 13168 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:25.902490 17273 ts_tablet_manager.cc:902] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:25.912932 17281 ts_tablet_manager.cc:902] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:25.921701 17280 tablet_replica.cc:330] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: stopping tablet replica
I20240503 02:13:25.926669 17280 raft_consensus.cc:2229] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:25.928957 17280 raft_consensus.cc:2258] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:25.940701 17288 ts_tablet_manager.cc:902] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:25.942662 17280 ts_tablet_manager.cc:1875] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:25.944296 12906 consensus_queue.cc:237] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:25.953588 10490 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:25.955648 10226 raft_consensus.cc:1261] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:25.956323 16984 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.002s
I20240503 02:13:25.958714 16730 consensus_queue.cc:1035] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:25.964624 17273 tablet_copy_client.cc:308] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:25.966077 16317 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:25.966833 10490 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:25.970952 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet 254af82e74a045f8a9c9d9de48cdee3e from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:58973)
I20240503 02:13:25.971213 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet 254af82e74a045f8a9c9d9de48cdee3e from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:58973: session id = 031a0afb01b6469299fa829fa3467ef7-254af82e74a045f8a9c9d9de48cdee3e
I20240503 02:13:25.975386 12928 tablet_copy_source_session.cc:205] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:25.987562 10227 raft_consensus.cc:2932] T 8c3effbb68eb4433bda2b2597ab1468a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:25.991140 17274 tablet_copy_client.cc:308] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:25.992172 17288 tablet_copy_client.cc:308] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:25.992813 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 8c3effbb68eb4433bda2b2597ab1468a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:25.993275 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet 9fbc1a14c000465e9bb642fcc1e84611 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:52841)
I20240503 02:13:25.993599 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet 9fbc1a14c000465e9bb642fcc1e84611 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:52841: session id = eff8e8d11a4447218741bb994aa41082-9fbc1a14c000465e9bb642fcc1e84611
I20240503 02:13:25.997718 17273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 254af82e74a045f8a9c9d9de48cdee3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.001777 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet ce47128b75a645e8a9f6e4cbc9d8d021 from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:36597)
I20240503 02:13:26.001951 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet ce47128b75a645e8a9f6e4cbc9d8d021 from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:36597: session id = c6ec2f1283994a3089741715f811c850-ce47128b75a645e8a9f6e4cbc9d8d021
I20240503 02:13:26.007030 10115 tablet_copy_source_session.cc:205] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.008735 12928 tablet_copy_source_session.cc:205] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.003304 9952 catalog_manager.cc:5484] T 8c3effbb68eb4433bda2b2597ab1468a P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 3 to 5, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 2 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:26.009994 17274 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ce47128b75a645e8a9f6e4cbc9d8d021. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.021260 17288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fbc1a14c000465e9bb642fcc1e84611. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.026955 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet 8c3effbb68eb4433bda2b2597ab1468a with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:26.033067 17273 tablet_copy_client.cc:783] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.034361 17273 tablet_copy_client.cc:647] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.038292 17274 tablet_copy_client.cc:783] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.038581 17274 tablet_copy_client.cc:647] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.038847 17280 ts_tablet_manager.cc:1888] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:26.039005 17280 log.cc:1196] T 13e83ce8820b491992c041a8c21fa3df P 031a0afb01b6469299fa829fa3467ef7: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-0/wal/wals/13e83ce8820b491992c041a8c21fa3df
I20240503 02:13:26.040123 9938 catalog_manager.cc:4830] TS 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): tablet 13e83ce8820b491992c041a8c21fa3df (table rebalance_test_table_1 [id=0ce87e63248d40ddb4c22ba5c8392b14]) successfully deleted
I20240503 02:13:26.041005 17273 tablet_copy_client.cc:523] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.042568 17274 tablet_copy_client.cc:523] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.046440 17274 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:26.054857 17273 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:26.062947 17307 ts_tablet_manager.cc:902] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:26.066561 17288 tablet_copy_client.cc:783] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.067095 17288 tablet_copy_client.cc:647] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.065332 17306 tablet_replica.cc:330] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:26.068053 17306 raft_consensus.cc:2229] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:26.069074 17306 raft_consensus.cc:2258] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:26.069965 17288 tablet_copy_client.cc:523] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.072168 17306 ts_tablet_manager.cc:1875] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:26.078511 17288 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:26.089004 17274 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.089416 17274 tablet_bootstrap.cc:492] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:26.089696 17274 ts_tablet_manager.cc:1367] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.007s
I20240503 02:13:26.090013 13172 consensus_queue.cc:237] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:26.090545 17274 raft_consensus.cc:348] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:26.090785 17274 raft_consensus.cc:729] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:26.091219 17274 consensus_queue.cc:260] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:26.092010 17274 ts_tablet_manager.cc:1398] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20240503 02:13:26.093015 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-ce47128b75a645e8a9f6e4cbc9d8d021 received from {username='slave'} at 127.4.5.68:36597
I20240503 02:13:26.093169 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session c6ec2f1283994a3089741715f811c850-ce47128b75a645e8a9f6e4cbc9d8d021 on tablet ce47128b75a645e8a9f6e4cbc9d8d021 with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:26.108340 17281 tablet_copy_client.cc:308] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
W20240503 02:13:26.111529 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bee269120a44b53b5bacc722f6742a9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.112011 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 4c014b2069ae42788aae87d8304fa155 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:35599)
I20240503 02:13:26.112257 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 4c014b2069ae42788aae87d8304fa155 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:35599: session id = 47f901e617024baaa1865d12261ef84f-4c014b2069ae42788aae87d8304fa155
I20240503 02:13:26.113103 13037 raft_consensus.cc:1261] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:26.113941 13295 raft_consensus.cc:1261] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:26.114009 17033 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:26.122216 10246 tablet_copy_source_session.cc:205] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.122566 17033 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:26.136224 17281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c014b2069ae42788aae87d8304fa155. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.144703 17288 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.145319 17288 tablet_bootstrap.cc:492] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:26.145685 17288 ts_tablet_manager.cc:1367] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.067s user 0.010s sys 0.000s
I20240503 02:13:26.146890 17288 raft_consensus.cc:348] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:26.147392 17288 raft_consensus.cc:729] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:26.147643 17288 consensus_queue.cc:260] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:26.159221 17288 ts_tablet_manager.cc:1398] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20240503 02:13:26.160612 17306 ts_tablet_manager.cc:1888] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 02:13:26.160018 17319 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:26.164202 17306 log.cc:1196] T 8c3effbb68eb4433bda2b2597ab1468a P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/8c3effbb68eb4433bda2b2597ab1468a
I20240503 02:13:26.166420 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 8c3effbb68eb4433bda2b2597ab1468a (table rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]) successfully deleted
I20240503 02:13:26.168884 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-9fbc1a14c000465e9bb642fcc1e84611 received from {username='slave'} at 127.4.5.66:52841
I20240503 02:13:26.169207 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session eff8e8d11a4447218741bb994aa41082-9fbc1a14c000465e9bb642fcc1e84611 on tablet 9fbc1a14c000465e9bb642fcc1e84611 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:26.172927 13037 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:26.173143 13295 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:26.175513 17273 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.176224 17273 tablet_bootstrap.cc:492] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:26.176537 17273 ts_tablet_manager.cc:1367] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.122s user 0.015s sys 0.016s
I20240503 02:13:26.180007 17273 raft_consensus.cc:348] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:26.180334 17273 raft_consensus.cc:729] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:26.181360 17273 consensus_queue.cc:260] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:26.182556 17273 ts_tablet_manager.cc:1398] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20240503 02:13:26.183516 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-254af82e74a045f8a9c9d9de48cdee3e received from {username='slave'} at 127.4.5.65:58973
I20240503 02:13:26.183727 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-254af82e74a045f8a9c9d9de48cdee3e on tablet 254af82e74a045f8a9c9d9de48cdee3e with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:26.185443 9951 catalog_manager.cc:5484] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:26.191148 17307 tablet_copy_client.cc:308] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:26.192176 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet fc0a049eeb0047558d3c9eb370467c53 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:60931)
I20240503 02:13:26.192360 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet fc0a049eeb0047558d3c9eb370467c53 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:60931: session id = f93b103af083471b8e8c0376e9148aad-fc0a049eeb0047558d3c9eb370467c53
I20240503 02:13:26.195482 17281 tablet_copy_client.cc:783] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.196008 17281 tablet_copy_client.cc:647] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.200219 17288 ts_tablet_manager.cc:902] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:26.201593 10115 tablet_copy_source_session.cc:205] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.203158 17281 tablet_copy_client.cc:523] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.207856 17281 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:26.208673 17307 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fc0a049eeb0047558d3c9eb370467c53. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.237164 17281 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.238149 17281 tablet_bootstrap.cc:492] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:26.238579 17281 ts_tablet_manager.cc:1367] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.031s user 0.008s sys 0.005s
I20240503 02:13:26.239652 17281 raft_consensus.cc:348] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:26.240106 17281 raft_consensus.cc:729] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:26.240612 17281 consensus_queue.cc:260] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:26.242646 17281 ts_tablet_manager.cc:1398] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:26.243846 17307 tablet_copy_client.cc:783] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.246285 17307 tablet_copy_client.cc:647] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.249161 17307 tablet_copy_client.cc:523] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.250478 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-4c014b2069ae42788aae87d8304fa155 received from {username='slave'} at 127.4.5.67:35599
I20240503 02:13:26.250638 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 47f901e617024baaa1865d12261ef84f-4c014b2069ae42788aae87d8304fa155 on tablet 4c014b2069ae42788aae87d8304fa155 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:26.268110 17307 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:26.283124 17273 ts_tablet_manager.cc:902] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:26.297955 17328 ts_tablet_manager.cc:902] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:26.321803 17288 tablet_copy_client.cc:308] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:26.323410 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet e4caac5d5607431ba8af43993e9de075 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:55635)
I20240503 02:13:26.323712 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet e4caac5d5607431ba8af43993e9de075 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:55635: session id = eff8e8d11a4447218741bb994aa41082-e4caac5d5607431ba8af43993e9de075
I20240503 02:13:26.327844 10115 tablet_copy_source_session.cc:205] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.330288 17288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e4caac5d5607431ba8af43993e9de075. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.339501 17288 tablet_copy_client.cc:783] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.340011 17288 tablet_copy_client.cc:647] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.347345 10359 consensus_queue.cc:237] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:26.349694 17288 tablet_copy_client.cc:523] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.356189 17288 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:26.361946 10490 raft_consensus.cc:1261] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:26.362957 12906 raft_consensus.cc:1261] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:26.363857 16707 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:26.364939 16919 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:26.377432 10227 raft_consensus.cc:1203] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:26.384974 10359 raft_consensus.cc:1203] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:26.400650 16707 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
W20240503 02:13:26.401415 10294 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c206318e0524101a4daee638813e041. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.413110 10490 raft_consensus.cc:1203] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:26.417829 10089 raft_consensus.cc:1203] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:26.435192 9952 catalog_manager.cc:5484] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:26.444151 10490 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:26.467739 12906 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:26.508862 17281 ts_tablet_manager.cc:902] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:26.509799 17328 tablet_copy_client.cc:308] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:26.511135 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet af7beeafa08f477796e5070fbc177dcc from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:38111)
I20240503 02:13:26.511341 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet af7beeafa08f477796e5070fbc177dcc from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:38111: session id = 8d454a3607d042dc9559ed59c3ccd84b-af7beeafa08f477796e5070fbc177dcc
I20240503 02:13:26.518224 10246 tablet_copy_source_session.cc:205] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.523213 17328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet af7beeafa08f477796e5070fbc177dcc. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.535813 17288 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.536356 17288 tablet_bootstrap.cc:492] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:26.536577 17273 tablet_copy_client.cc:308] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:13:26.536840 17288 ts_tablet_manager.cc:1367] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.181s user 0.050s sys 0.049s
I20240503 02:13:26.536840 17328 tablet_copy_client.cc:783] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.537353 17328 tablet_copy_client.cc:647] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.537881 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet f385792ed7c8465eb7cd64142778c9bf from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:56937)
I20240503 02:13:26.538074 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet f385792ed7c8465eb7cd64142778c9bf from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:56937: session id = 031a0afb01b6469299fa829fa3467ef7-f385792ed7c8465eb7cd64142778c9bf
I20240503 02:13:26.538061 17288 raft_consensus.cc:348] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:26.538450 17288 raft_consensus.cc:729] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:26.538769 17288 consensus_queue.cc:260] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:26.542869 17288 ts_tablet_manager.cc:1398] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20240503 02:13:26.543802 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-e4caac5d5607431ba8af43993e9de075 received from {username='slave'} at 127.4.5.66:55635
I20240503 02:13:26.543967 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session eff8e8d11a4447218741bb994aa41082-e4caac5d5607431ba8af43993e9de075 on tablet e4caac5d5607431ba8af43993e9de075 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:26.546823 10511 tablet_copy_source_session.cc:205] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.551534 17328 tablet_copy_client.cc:523] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.551935 17273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f385792ed7c8465eb7cd64142778c9bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.576196 17328 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:26.613087 12906 consensus_queue.cc:237] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
W20240503 02:13:26.623613 12840 consensus_peers.cc:485] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d856c707315b47e996ee604a5ff66b31. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:26.624117 13168 raft_consensus.cc:1261] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:26.624188 13040 raft_consensus.cc:1261] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:26.625046 16317 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:26.625768 16317 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:26.646273 10227 raft_consensus.cc:1203] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:26.679513 16984 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:26.684263 13036 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:26.694561 9952 catalog_manager.cc:5484] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:26.709120 17273 tablet_copy_client.cc:783] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.709877 17273 tablet_copy_client.cc:647] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.726461 17273 tablet_copy_client.cc:523] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.742462 17273 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:26.747030 17281 tablet_copy_client.cc:308] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.71:46557
I20240503 02:13:26.749881 13195 tablet_copy_service.cc:140] P f93b103af083471b8e8c0376e9148aad: Received BeginTabletCopySession request for tablet 0bee269120a44b53b5bacc722f6742a9 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:51149)
I20240503 02:13:26.750155 13195 tablet_copy_service.cc:161] P f93b103af083471b8e8c0376e9148aad: Beginning new tablet copy session on tablet 0bee269120a44b53b5bacc722f6742a9 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:51149: session id = 47f901e617024baaa1865d12261ef84f-0bee269120a44b53b5bacc722f6742a9
I20240503 02:13:26.752599 13166 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:26.758214 13195 tablet_copy_source_session.cc:205] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.760397 17281 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0bee269120a44b53b5bacc722f6742a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.770927 17281 tablet_copy_client.cc:783] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:26.773597 17281 tablet_copy_client.cc:647] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:26.779174 16730 raft_consensus.cc:1050] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:26.780253 16730 consensus_queue.cc:237] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:26.834580 17273 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:26.837929 17273 tablet_bootstrap.cc:492] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:26.838474 17273 ts_tablet_manager.cc:1367] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.096s user 0.009s sys 0.001s
I20240503 02:13:26.841867 17273 raft_consensus.cc:348] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:26.848556 17273 raft_consensus.cc:729] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:26.849074 17273 consensus_queue.cc:260] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:26.850260 17273 ts_tablet_manager.cc:1398] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20240503 02:13:26.874878 17281 tablet_copy_client.cc:523] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:26.879698 17281 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:26.887501 17374 ts_tablet_manager.cc:902] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:26.892235 10227 raft_consensus.cc:1261] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:26.894338 10359 raft_consensus.cc:1261] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:26.905586 16984 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:26.910871 17378 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:26.912693 17378 raft_consensus.cc:1050] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:26.917572 17378 consensus_queue.cc:237] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:26.920837 17374 tablet_copy_client.cc:308] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:26.922108 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 4c206318e0524101a4daee638813e041 from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:32791)
I20240503 02:13:26.922394 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 4c206318e0524101a4daee638813e041 from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:32791: session id = 031a0afb01b6469299fa829fa3467ef7-4c206318e0524101a4daee638813e041
I20240503 02:13:26.925871 10380 tablet_copy_source_session.cc:205] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:26.927884 17374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c206318e0524101a4daee638813e041. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:26.935737 16501 raft_consensus.cc:1050] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:26.937208 16501 consensus_queue.cc:237] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
W20240503 02:13:26.967900 12842 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.66:42397, user_credentials={real_user=slave}} blocked reactor thread for 66334.3us
I20240503 02:13:26.977545 13295 raft_consensus.cc:1261] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:26.985729 17372 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:26.989832 10089 raft_consensus.cc:1203] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:26.993584 10359 raft_consensus.cc:1261] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:26.994827 16941 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.001000 10089 raft_consensus.cc:1261] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.000977 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-f385792ed7c8465eb7cd64142778c9bf received from {username='slave'} at 127.4.5.65:56937
I20240503 02:13:27.001173 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-f385792ed7c8465eb7cd64142778c9bf on tablet f385792ed7c8465eb7cd64142778c9bf with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:27.002296 16765 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.005184 17222 raft_consensus.cc:1050] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:27.006321 17222 consensus_queue.cc:237] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:27.022279 13166 raft_consensus.cc:1261] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.024315 17358 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.026918 17374 tablet_copy_client.cc:783] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:27.028195 13040 raft_consensus.cc:1261] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.032248 16941 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:27.035496 10490 raft_consensus.cc:1261] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.035938 13295 raft_consensus.cc:1261] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240503 02:13:27.055632 10159 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.68:35025, user_credentials={real_user=slave}} blocked reactor thread for 48922.1us
I20240503 02:13:27.057134 10359 raft_consensus.cc:1261] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.057893 17378 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.058573 13295 raft_consensus.cc:1261] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240503 02:13:27.058888 17307 log.cc:924] Time spent T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: Append to log took a long time: real 0.064s user 0.001s sys 0.000s
I20240503 02:13:27.061964 16765 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.065352 10359 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:27.066313 9951 catalog_manager.cc:5484] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:27.068957 17372 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.076138 10226 consensus_queue.cc:237] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:27.080829 9951 catalog_manager.cc:5484] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:27.039783 16654 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.054998 17222 raft_consensus.cc:1050] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:27.087563 17222 consensus_queue.cc:237] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:27.088523 17358 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.090822 17374 tablet_copy_client.cc:647] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:27.055812 10089 raft_consensus.cc:1261] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.065361 10091 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:27.098354 10092 raft_consensus.cc:1261] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:27.098970 10353 raft_consensus.cc:1261] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:27.100054 17393 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:27.105274 10353 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:27.111325 16501 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:27.121340 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c014b2069ae42788aae87d8304fa155 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 02:13:27.127802 12842 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.66:42397, user_credentials={real_user=slave}} blocked reactor thread for 49153.7us
I20240503 02:13:27.129163 10359 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.134752 13302 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.153628 17373 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.154660 10226 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.155655 9947 catalog_manager.cc:5484] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:27.172519 17392 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:27.171730 17394 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:27.177413 10490 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:27.179396 13302 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:27.183269 10226 raft_consensus.cc:1261] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.184741 17373 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.186730 12908 consensus_queue.cc:237] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:27.193320 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c014b2069ae42788aae87d8304fa155 with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:27.198432 9943 catalog_manager.cc:5484] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:27.207803 13166 raft_consensus.cc:1261] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.211515 17307 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.215919 13040 raft_consensus.cc:1261] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.221369 10490 consensus_queue.cc:237] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:27.231740 17281 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.237736 13305 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:27.239956 10089 consensus_queue.cc:237] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:27.243067 17405 ts_tablet_manager.cc:902] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:27.244098 10491 raft_consensus.cc:1261] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:27.244935 13295 raft_consensus.cc:1261] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:27.246564 10226 raft_consensus.cc:1261] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:27.248611 17281 tablet_bootstrap.cc:492] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:27.253011 17397 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:27.253994 17397 consensus_queue.cc:1035] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:27.257701 17307 tablet_bootstrap.cc:492] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
W20240503 02:13:27.258947 10426 consensus_peers.cc:485] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 727b42f4ac0f45d9bf7c445f99fc3630. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:27.262099 17371 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.264001 17394 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:27.267513 13302 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:27.269701 10490 raft_consensus.cc:2932] T ce47128b75a645e8a9f6e4cbc9d8d021 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:27.277897 9952 catalog_manager.cc:5484] T ce47128b75a645e8a9f6e4cbc9d8d021 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:27.281126 10091 raft_consensus.cc:1261] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:27.285403 17236 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:27.289382 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ce47128b75a645e8a9f6e4cbc9d8d021 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:27.293640 10092 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:27.295295 17328 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.296013 17328 tablet_bootstrap.cc:492] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:27.296473 17328 ts_tablet_manager.cc:1367] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.720s user 0.071s sys 0.019s
I20240503 02:13:27.298131 13040 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.250531 10353 raft_consensus.cc:1261] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:27.310250 9951 catalog_manager.cc:5484] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:27.311368 17328 raft_consensus.cc:348] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:27.314049 17328 raft_consensus.cc:729] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:27.314426 17328 consensus_queue.cc:260] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:27.323171 17281 ts_tablet_manager.cc:1367] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.444s user 0.013s sys 0.002s
I20240503 02:13:27.324983 17281 raft_consensus.cc:348] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:27.325891 17281 raft_consensus.cc:729] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:27.326534 17328 ts_tablet_manager.cc:1398] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20240503 02:13:27.327647 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-af7beeafa08f477796e5070fbc177dcc received from {username='slave'} at 127.4.5.70:38111
I20240503 02:13:27.327634 17394 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.327880 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-af7beeafa08f477796e5070fbc177dcc on tablet af7beeafa08f477796e5070fbc177dcc with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:27.330055 17394 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.337121 13036 raft_consensus.cc:1203] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:27.264292 13166 raft_consensus.cc:1261] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:27.285434 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet ce47128b75a645e8a9f6e4cbc9d8d021 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:27.351544 17417 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:27.370988 17307 ts_tablet_manager.cc:1367] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 1.103s user 0.065s sys 0.018s
I20240503 02:13:27.374794 17421 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:27.382774 17420 raft_consensus.cc:1050] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:27.390812 10091 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
W20240503 02:13:27.406132 12842 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 145292us
I20240503 02:13:27.411201 17420 consensus_queue.cc:237] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:27.413919 17373 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:27.418746 16984 consensus_queue.cc:1035] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:27.424986 17397 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.425904 17374 tablet_copy_client.cc:523] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:27.430536 10089 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.431237 17374 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:27.431257 17412 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.434360 13040 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
W20240503 02:13:27.436254 17395 log.cc:924] Time spent T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Append to log took a long time: real 0.053s user 0.000s sys 0.000s
I20240503 02:13:27.438724 13305 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:27.439563 17412 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:27.375296 17416 tablet_replica.cc:330] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:27.382465 13166 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:27.444721 17416 raft_consensus.cc:2229] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:27.446741 17416 pending_rounds.cc:62] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: Trying to abort 1 pending ops.
I20240503 02:13:27.447140 17416 pending_rounds.cc:68] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023421059094904832 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "ce47128b75a645e8a9f6e4cbc9d8d021" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } } }
I20240503 02:13:27.448102 17416 raft_consensus.cc:2894] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 02:13:27.446280 17420 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.002s
I20240503 02:13:27.448478 17416 raft_consensus.cc:2866] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:27.448715 17416 raft_consensus.cc:2258] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.452941 17374 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.453366 17416 ts_tablet_manager.cc:1875] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:27.453896 17374 tablet_bootstrap.cc:492] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:27.454341 17374 ts_tablet_manager.cc:1367] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.023s user 0.013s sys 0.002s
I20240503 02:13:27.455392 17374 raft_consensus.cc:348] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:27.455715 17374 raft_consensus.cc:729] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:27.458250 9952 catalog_manager.cc:5484] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:13:27.456102 17374 consensus_queue.cc:260] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:27.467212 17374 ts_tablet_manager.cc:1398] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.013s user 0.004s sys 0.000s
I20240503 02:13:27.479065 17412 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.480278 10092 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.481535 17404 tablet_replica.cc:330] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:27.487224 17404 raft_consensus.cc:2229] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:13:27.486588 10028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 60375.2us
I20240503 02:13:27.498553 9951 catalog_manager.cc:5484] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.503597 17404 raft_consensus.cc:2258] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.512853 17404 ts_tablet_manager.cc:1875] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:13:27.515883 12842 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 97333.7us
I20240503 02:13:27.445809 17307 raft_consensus.cc:348] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:27.517441 17307 raft_consensus.cc:729] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:27.517829 17307 consensus_queue.cc:260] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:27.518659 17307 ts_tablet_manager.cc:1398] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.075s user 0.000s sys 0.002s
I20240503 02:13:27.520956 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-fc0a049eeb0047558d3c9eb370467c53 received from {username='slave'} at 127.4.5.71:60931
I20240503 02:13:27.521106 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session f93b103af083471b8e8c0376e9148aad-fc0a049eeb0047558d3c9eb370467c53 on tablet fc0a049eeb0047558d3c9eb370467c53 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:27.530988 17371 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.326368 17281 consensus_queue.cc:260] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:27.540140 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9fbc1a14c000465e9bb642fcc1e84611 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:27.548588 10359 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.557684 10224 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.569082 17281 ts_tablet_manager.cc:1398] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.246s user 0.005s sys 0.000s
I20240503 02:13:27.574365 13040 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.580891 17416 ts_tablet_manager.cc:1888] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:27.600718 17416 log.cc:1196] T ce47128b75a645e8a9f6e4cbc9d8d021 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/ce47128b75a645e8a9f6e4cbc9d8d021
I20240503 02:13:27.602047 9952 catalog_manager.cc:5484] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:27.602406 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet ce47128b75a645e8a9f6e4cbc9d8d021 (table rebalancer_extra_table_18 [id=4a225e182e2b47cb99a273c7198b1dc3]) successfully deleted
I20240503 02:13:27.610621 12908 consensus_queue.cc:237] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:27.618403 17405 tablet_copy_client.cc:308] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.69:33471
I20240503 02:13:27.642690 10359 raft_consensus.cc:1261] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:27.653110 12928 tablet_copy_service.cc:140] P e3cbe34992dc4b77b9aa9e9a4c143e78: Received BeginTabletCopySession request for tablet d856c707315b47e996ee604a5ff66b31 from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:35461)
I20240503 02:13:27.653465 12928 tablet_copy_service.cc:161] P e3cbe34992dc4b77b9aa9e9a4c143e78: Beginning new tablet copy session on tablet d856c707315b47e996ee604a5ff66b31 from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:35461: session id = c6ec2f1283994a3089741715f811c850-d856c707315b47e996ee604a5ff66b31
I20240503 02:13:27.664402 12928 tablet_copy_source_session.cc:205] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78: Tablet Copy: opened 0 blocks and 1 log segments
W20240503 02:13:27.665866 10292 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 291084us
I20240503 02:13:27.679745 10355 raft_consensus.cc:2932] T 9fbc1a14c000465e9bb642fcc1e84611 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.686383 17405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d856c707315b47e996ee604a5ff66b31. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:27.703404 17405 tablet_copy_client.cc:783] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:27.703963 17405 tablet_copy_client.cc:647] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:27.704509 16984 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:27.713322 17373 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.719159 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-4c206318e0524101a4daee638813e041 received from {username='slave'} at 127.4.5.65:32791
I20240503 02:13:27.719406 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-4c206318e0524101a4daee638813e041 on tablet 4c206318e0524101a4daee638813e041 with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:27.720597 10358 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.726073 10353 raft_consensus.cc:1203] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:27.729518 10091 raft_consensus.cc:1203] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:27.730459 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 254af82e74a045f8a9c9d9de48cdee3e with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:27.731863 17405 tablet_copy_client.cc:523] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:27.733861 9951 catalog_manager.cc:5484] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:27.737164 17405 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:27.739985 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet 254af82e74a045f8a9c9d9de48cdee3e with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52066
W20240503 02:13:27.769665 10029 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.70:43707, user_credentials={real_user=slave}} blocked reactor thread for 256303us
I20240503 02:13:27.792702 17394 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.813686 9951 catalog_manager.cc:5484] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:27.826707 17442 tablet_replica.cc:330] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:27.827242 17442 raft_consensus.cc:2229] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
W20240503 02:13:27.828313 17428 log.cc:924] Time spent T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Append to log took a long time: real 0.255s user 0.000s sys 0.000s
I20240503 02:13:27.828347 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fbc1a14c000465e9bb642fcc1e84611 with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:27.829524 17442 raft_consensus.cc:2258] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:27.842777 13305 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.869681 13195 tablet_copy_service.cc:342] P f93b103af083471b8e8c0376e9148aad: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-0bee269120a44b53b5bacc722f6742a9 received from {username='slave'} at 127.4.5.67:51149
I20240503 02:13:27.869971 13195 tablet_copy_service.cc:433] P f93b103af083471b8e8c0376e9148aad: ending tablet copy session 47f901e617024baaa1865d12261ef84f-0bee269120a44b53b5bacc722f6742a9 on tablet 0bee269120a44b53b5bacc722f6742a9 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:27.899220 13168 consensus_queue.cc:237] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:27.899533 17405 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:27.900089 17405 tablet_bootstrap.cc:492] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:27.900404 17405 ts_tablet_manager.cc:1367] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.163s user 0.036s sys 0.066s
I20240503 02:13:27.901446 17405 raft_consensus.cc:348] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:27.901803 17405 raft_consensus.cc:729] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:27.902058 17405 consensus_queue.cc:260] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:27.902840 17405 ts_tablet_manager.cc:1398] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:27.908785 12928 tablet_copy_service.cc:342] P e3cbe34992dc4b77b9aa9e9a4c143e78: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-d856c707315b47e996ee604a5ff66b31 received from {username='slave'} at 127.4.5.68:35461
I20240503 02:13:27.909071 12928 tablet_copy_service.cc:433] P e3cbe34992dc4b77b9aa9e9a4c143e78: ending tablet copy session c6ec2f1283994a3089741715f811c850-d856c707315b47e996ee604a5ff66b31 on tablet d856c707315b47e996ee604a5ff66b31 with peer c6ec2f1283994a3089741715f811c850
W20240503 02:13:27.913753 13107 consensus_peers.cc:485] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 7afcfd9776ff4097a749ab8f91730d7b. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:27.920264 13305 raft_consensus.cc:1261] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:27.921157 17424 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:27.921376 13166 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:27.933537 10091 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:27.940044 17447 ts_tablet_manager.cc:902] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
W20240503 02:13:27.941792 10028 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 69294.4us
I20240503 02:13:27.831187 17442 ts_tablet_manager.cc:1875] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:13:27.946367 12976 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 63672.1us
I20240503 02:13:27.911692 13040 raft_consensus.cc:1261] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:27.958746 17389 raft_consensus.cc:1050] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:27.959996 17442 ts_tablet_manager.cc:1888] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:27.961071 17442 log.cc:1196] T 254af82e74a045f8a9c9d9de48cdee3e P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/254af82e74a045f8a9c9d9de48cdee3e
I20240503 02:13:27.961583 17447 tablet_copy_client.cc:308] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:13:27.962883 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet 254af82e74a045f8a9c9d9de48cdee3e (table rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]) successfully deleted
I20240503 02:13:27.965245 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet 727b42f4ac0f45d9bf7c445f99fc3630 from peer eff8e8d11a4447218741bb994aa41082 ({username='slave'} at 127.4.5.66:51843)
I20240503 02:13:27.965420 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet 727b42f4ac0f45d9bf7c445f99fc3630 from peer eff8e8d11a4447218741bb994aa41082 at {username='slave'} at 127.4.5.66:51843: session id = eff8e8d11a4447218741bb994aa41082-727b42f4ac0f45d9bf7c445f99fc3630
I20240503 02:13:27.965834 17424 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:27.960285 17389 consensus_queue.cc:237] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:27.968843 10511 tablet_copy_source_session.cc:205] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:27.976012 17321 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:27.977592 13040 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:27.980656 17318 raft_consensus.cc:1050] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:27.981736 17318 consensus_queue.cc:237] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:27.995626 13166 raft_consensus.cc:1203] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:27.995728 9951 catalog_manager.cc:5484] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:27.997009 10091 raft_consensus.cc:1261] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.000643 10091 consensus_queue.cc:237] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:28.006779 17404 ts_tablet_manager.cc:1888] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:28.007155 17373 consensus_queue.cc:1035] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.009039 13040 raft_consensus.cc:1261] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.011806 10091 raft_consensus.cc:2932] T 254af82e74a045f8a9c9d9de48cdee3e P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.018648 17358 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.030058 10359 raft_consensus.cc:1261] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.030189 13037 raft_consensus.cc:1261] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.031387 13040 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:28.033738 17424 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.034559 17424 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.038936 17447 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 727b42f4ac0f45d9bf7c445f99fc3630. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.039296 10227 raft_consensus.cc:1261] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.040500 17448 consensus_queue.cc:1035] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.045588 12908 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.048485 13036 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.056532 10224 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:28.033367 13302 raft_consensus.cc:1261] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.064064 17318 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.060945 13305 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:28.067036 17393 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.018s
I20240503 02:13:28.069689 17404 log.cc:1196] T 4c014b2069ae42788aae87d8304fa155 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/4c014b2069ae42788aae87d8304fa155
I20240503 02:13:28.069698 17392 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:28.082674 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 4c014b2069ae42788aae87d8304fa155 (table rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]) successfully deleted
I20240503 02:13:28.084322 17447 tablet_copy_client.cc:783] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:28.085232 17404 tablet_replica.cc:330] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:28.085240 17447 tablet_copy_client.cc:647] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:28.089365 17404 raft_consensus.cc:2229] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.092131 17447 tablet_copy_client.cc:523] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:28.096515 17404 raft_consensus.cc:2258] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.098452 10355 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.099526 17393 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:28.105656 17447 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: Bootstrap starting.
I20240503 02:13:28.119181 17448 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:28.120877 10227 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:28.133499 9951 catalog_manager.cc:5484] T e4caac5d5607431ba8af43993e9de075 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:28.135864 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e4caac5d5607431ba8af43993e9de075 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.139025 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet e4caac5d5607431ba8af43993e9de075 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:28.143899 10491 raft_consensus.cc:1203] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:28.150569 17321 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:28.156898 13302 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:28.157990 13036 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
W20240503 02:13:28.159415 10029 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.70:43707, user_credentials={real_user=slave}} blocked reactor thread for 91490.5us
I20240503 02:13:28.166818 17436 raft_consensus.cc:1050] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:28.168052 17436 consensus_queue.cc:237] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:28.170861 9952 catalog_manager.cc:5484] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:28.179697 17447 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:28.188694 17447 tablet_bootstrap.cc:492] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: Bootstrap complete.
I20240503 02:13:28.189236 17447 ts_tablet_manager.cc:1367] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: Time spent bootstrapping tablet: real 0.084s user 0.012s sys 0.013s
I20240503 02:13:28.190249 17447 raft_consensus.cc:348] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:28.190622 17447 raft_consensus.cc:729] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Initialized, Role: LEARNER
I20240503 02:13:28.190887 17447 consensus_queue.cc:260] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:13:28.196286 17447 ts_tablet_manager.cc:1398] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20240503 02:13:28.197362 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session eff8e8d11a4447218741bb994aa41082-727b42f4ac0f45d9bf7c445f99fc3630 received from {username='slave'} at 127.4.5.66:51843
I20240503 02:13:28.197527 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session eff8e8d11a4447218741bb994aa41082-727b42f4ac0f45d9bf7c445f99fc3630 on tablet 727b42f4ac0f45d9bf7c445f99fc3630 with peer eff8e8d11a4447218741bb994aa41082
I20240503 02:13:28.201224 17404 ts_tablet_manager.cc:1875] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.215596 10091 raft_consensus.cc:1261] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.216488 17436 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.218894 10491 raft_consensus.cc:1261] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.219090 12906 raft_consensus.cc:1261] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.225524 17437 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.228087 17437 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.228893 17464 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.233855 13037 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.240914 12906 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.241818 10355 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.253629 9951 catalog_manager.cc:5484] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.265960 10227 consensus_queue.cc:237] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:28.268321 10487 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.266836 17436 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.278784 13037 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:28.280560 17465 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:28.281857 9947 catalog_manager.cc:5484] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.288012 17464 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.291160 17471 tablet_replica.cc:330] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:28.291574 17471 raft_consensus.cc:2229] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.292048 17471 raft_consensus.cc:2258] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.292922 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af7beeafa08f477796e5070fbc177dcc with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.293949 10226 raft_consensus.cc:1203] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:28.296831 13036 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.302148 10359 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:28.310390 9951 catalog_manager.cc:5484] T af7beeafa08f477796e5070fbc177dcc P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.318121 13166 consensus_queue.cc:237] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:28.320662 10355 raft_consensus.cc:1261] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.320817 12906 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.321463 17321 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.323866 12908 raft_consensus.cc:1261] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:28.325037 17321 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:28.328508 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet af7beeafa08f477796e5070fbc177dcc with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:28.329238 10355 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:28.330053 13040 raft_consensus.cc:1261] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.330855 17393 consensus_queue.cc:1035] T af7beeafa08f477796e5070fbc177dcc P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:28.333397 17459 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.337003 12906 raft_consensus.cc:2932] T af7beeafa08f477796e5070fbc177dcc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:28.340418 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0bee269120a44b53b5bacc722f6742a9 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.340646 13040 raft_consensus.cc:2932] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:28.320297 10091 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.353287 13040 raft_consensus.cc:2932] T e4caac5d5607431ba8af43993e9de075 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:28.358274 17404 ts_tablet_manager.cc:1888] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:28.358525 17404 log.cc:1196] T 9fbc1a14c000465e9bb642fcc1e84611 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/9fbc1a14c000465e9bb642fcc1e84611
I20240503 02:13:28.359243 17471 ts_tablet_manager.cc:1875] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.359490 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 9fbc1a14c000465e9bb642fcc1e84611 (table rebalancer_extra_table_20 [id=677911280e6f46edbbdc5cda886b9f0d]) successfully deleted
I20240503 02:13:28.364476 9947 catalog_manager.cc:5484] T 0bee269120a44b53b5bacc722f6742a9 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:28.368326 17477 tablet_replica.cc:330] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:28.368798 17477 raft_consensus.cc:2229] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.369275 17477 raft_consensus.cc:2258] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.383571 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 0bee269120a44b53b5bacc722f6742a9 with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:28.385167 10359 consensus_queue.cc:237] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:28.386431 17405 ts_tablet_manager.cc:902] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:28.388052 10091 raft_consensus.cc:1261] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.388180 10491 raft_consensus.cc:1261] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.389228 17438 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.389670 17436 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.397390 17477 ts_tablet_manager.cc:1875] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.400843 17404 tablet_replica.cc:330] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:28.401352 17404 raft_consensus.cc:2229] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.401839 17404 raft_consensus.cc:2258] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.404083 17397 raft_consensus.cc:1050] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:28.405337 17397 consensus_queue.cc:237] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:28.409411 17404 ts_tablet_manager.cc:1875] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.412868 17437 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.413846 10089 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.422451 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 4c206318e0524101a4daee638813e041 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.422690 10491 raft_consensus.cc:2932] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.435825 9943 catalog_manager.cc:5484] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:28.441691 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c206318e0524101a4daee638813e041 with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:28.465322 17404 ts_tablet_manager.cc:1888] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:28.470851 17485 tablet_replica.cc:330] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:28.471356 17485 raft_consensus.cc:2229] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.479270 17477 ts_tablet_manager.cc:1888] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:28.479580 17477 log.cc:1196] T af7beeafa08f477796e5070fbc177dcc P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/af7beeafa08f477796e5070fbc177dcc
I20240503 02:13:28.481381 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet af7beeafa08f477796e5070fbc177dcc (table rebalance_test_table_9 [id=4a9dc81443524db5a8a83e5f371bfde9]) successfully deleted
I20240503 02:13:28.483755 17485 raft_consensus.cc:2258] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.489595 17485 ts_tablet_manager.cc:1875] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.491732 17471 ts_tablet_manager.cc:1888] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:28.496623 17471 log.cc:1196] T e4caac5d5607431ba8af43993e9de075 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/e4caac5d5607431ba8af43993e9de075
I20240503 02:13:28.499282 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet e4caac5d5607431ba8af43993e9de075 (table rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]) successfully deleted
I20240503 02:13:28.506659 13040 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.507740 17448 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:28.516600 13166 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.517530 17448 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:28.520750 17404 log.cc:1196] T 0bee269120a44b53b5bacc722f6742a9 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/0bee269120a44b53b5bacc722f6742a9
I20240503 02:13:28.522147 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 0bee269120a44b53b5bacc722f6742a9 (table rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]) successfully deleted
I20240503 02:13:28.533516 10487 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:28.534528 17448 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:28.555981 17448 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:28.564465 10487 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:28.566269 13166 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:28.578795 9943 catalog_manager.cc:5484] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:28.600922 10227 consensus_queue.cc:237] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:28.612010 13040 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:28.614492 10091 consensus_queue.cc:237] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:28.624815 13168 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:28.626195 17396 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:28.631682 17396 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:28.637960 13168 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:28.645699 13036 raft_consensus.cc:1261] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:28.652915 17397 consensus_queue.cc:1035] T fc0a049eeb0047558d3c9eb370467c53 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:28.655267 13037 raft_consensus.cc:2932] T fc0a049eeb0047558d3c9eb370467c53 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:28.660965 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet fc0a049eeb0047558d3c9eb370467c53 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
W20240503 02:13:28.662405 10159 consensus_peers.cc:485] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5aa7ed62ebf44431a1a6db26ef73052a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:28.664000 10359 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.669384 17393 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.669580 17405 tablet_copy_client.cc:308] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.71:46557
I20240503 02:13:28.672255 13195 tablet_copy_service.cc:140] P f93b103af083471b8e8c0376e9148aad: Received BeginTabletCopySession request for tablet 7afcfd9776ff4097a749ab8f91730d7b from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:36981)
I20240503 02:13:28.676671 13195 tablet_copy_service.cc:161] P f93b103af083471b8e8c0376e9148aad: Beginning new tablet copy session on tablet 7afcfd9776ff4097a749ab8f91730d7b from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:36981: session id = c6ec2f1283994a3089741715f811c850-7afcfd9776ff4097a749ab8f91730d7b
I20240503 02:13:28.677165 13166 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:28.681988 17465 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:28.683095 13195 tablet_copy_source_session.cc:205] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:28.685523 17405 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7afcfd9776ff4097a749ab8f91730d7b. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:28.710148 17405 tablet_copy_client.cc:783] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:28.710712 17405 tablet_copy_client.cc:647] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:28.713374 17405 tablet_copy_client.cc:523] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:28.713464 17485 ts_tablet_manager.cc:1888] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:28.717453 9943 catalog_manager.cc:5484] T fc0a049eeb0047558d3c9eb370467c53 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 8 to 9, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:28.718257 17405 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:28.729641 10470 tablet_service.cc:1514] Processing DeleteTablet for tablet fc0a049eeb0047558d3c9eb370467c53 with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:28.748629 17495 tablet_replica.cc:330] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:28.749672 17405 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:28.750241 17405 tablet_bootstrap.cc:492] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:28.750622 17405 ts_tablet_manager.cc:1367] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.032s user 0.020s sys 0.010s
I20240503 02:13:28.751593 17405 raft_consensus.cc:348] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:28.751920 17405 raft_consensus.cc:729] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:28.752158 17405 consensus_queue.cc:260] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:28.750241 17495 raft_consensus.cc:2229] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:28.753789 17495 raft_consensus.cc:2258] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:28.754858 17405 ts_tablet_manager.cc:1398] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:28.764817 13195 tablet_copy_service.cc:342] P f93b103af083471b8e8c0376e9148aad: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-7afcfd9776ff4097a749ab8f91730d7b received from {username='slave'} at 127.4.5.68:36981
I20240503 02:13:28.765084 13195 tablet_copy_service.cc:433] P f93b103af083471b8e8c0376e9148aad: ending tablet copy session c6ec2f1283994a3089741715f811c850-7afcfd9776ff4097a749ab8f91730d7b on tablet 7afcfd9776ff4097a749ab8f91730d7b with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:28.777014 17495 ts_tablet_manager.cc:1875] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:28.763049 16984 raft_consensus.cc:1050] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:28.793965 17485 log.cc:1196] T 4c206318e0524101a4daee638813e041 P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/4c206318e0524101a4daee638813e041
I20240503 02:13:28.794951 16984 consensus_queue.cc:237] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:28.796690 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet 4c206318e0524101a4daee638813e041 (table rebalancer_extra_table_24 [id=e1a0b86c40a44543a8269ad2d2067584]) successfully deleted
I20240503 02:13:28.797890 17393 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:28.801987 10355 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:28.818723 9943 catalog_manager.cc:5484] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:28.819736 13168 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:28.827008 17495 ts_tablet_manager.cc:1888] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:28.827270 17495 log.cc:1196] T fc0a049eeb0047558d3c9eb370467c53 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/fc0a049eeb0047558d3c9eb370467c53
I20240503 02:13:28.833947 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet fc0a049eeb0047558d3c9eb370467c53 (table rebalance_test_table_11 [id=479a48d03d45493299d5086553ba9b8b]) successfully deleted
I20240503 02:13:28.853724 17418 raft_consensus.cc:1050] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER eff8e8d11a4447218741bb994aa41082 to VOTER
I20240503 02:13:28.854813 17418 consensus_queue.cc:237] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:28.876480 13168 raft_consensus.cc:1261] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.877023 13037 raft_consensus.cc:1261] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.877449 17373 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.878263 17373 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.878487 10487 raft_consensus.cc:1261] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.879544 17371 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.893548 10227 raft_consensus.cc:1261] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.894006 10091 raft_consensus.cc:1261] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.894862 13174 raft_consensus.cc:1261] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:28.895372 17418 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.895980 17417 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.900298 17373 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:28.908552 13166 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:28.911698 9951 catalog_manager.cc:5484] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:28.914155 13037 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:28.906996 10487 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:28.925057 12906 consensus_queue.cc:237] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:28.929234 17418 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:28.938529 13166 raft_consensus.cc:1261] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.942618 17373 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.946552 10487 raft_consensus.cc:1261] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer e3cbe34992dc4b77b9aa9e9a4c143e78: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:28.950162 17501 consensus_queue.cc:1035] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:28.951278 17373 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:28.961568 10490 raft_consensus.cc:1203] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:28.963862 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet d856c707315b47e996ee604a5ff66b31 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:28.965253 10490 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:28.965409 13166 raft_consensus.cc:2932] T d856c707315b47e996ee604a5ff66b31 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:28.974160 9951 catalog_manager.cc:5484] T d856c707315b47e996ee604a5ff66b31 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 1 leader_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:28.985874 17498 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:28.992718 13166 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:28.993324 10226 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:29.001063 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet d856c707315b47e996ee604a5ff66b31 with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52066
I20240503 02:13:29.004492 9944 catalog_manager.cc:5484] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:29.018985 10091 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, eff8e8d11a4447218741bb994aa41082 (127.4.5.66) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:29.026429 10487 consensus_queue.cc:237] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } }
I20240503 02:13:29.030426 10089 raft_consensus.cc:1261] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:29.031519 17418 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.035519 17498 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:29.045260 10089 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:29.051510 10227 raft_consensus.cc:1261] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:29.055147 17418 consensus_queue.cc:1035] T 727b42f4ac0f45d9bf7c445f99fc3630 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.058245 10226 raft_consensus.cc:2932] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:29.068931 9951 catalog_manager.cc:5484] T 727b42f4ac0f45d9bf7c445f99fc3630 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } }
I20240503 02:13:29.075029 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet 727b42f4ac0f45d9bf7c445f99fc3630 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:29.089226 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 727b42f4ac0f45d9bf7c445f99fc3630 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:29.103582 17522 tablet_replica.cc:330] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:29.103950 17522 raft_consensus.cc:2229] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:29.104318 17522 raft_consensus.cc:2258] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:29.105234 17515 tablet_replica.cc:330] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:29.105578 17515 raft_consensus.cc:2229] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:29.110311 17515 raft_consensus.cc:2258] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:29.112270 17522 ts_tablet_manager.cc:1875] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:29.113205 17515 ts_tablet_manager.cc:1875] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:29.122876 17522 ts_tablet_manager.cc:1888] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:29.123175 17522 log.cc:1196] T 727b42f4ac0f45d9bf7c445f99fc3630 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/727b42f4ac0f45d9bf7c445f99fc3630
I20240503 02:13:29.124246 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 727b42f4ac0f45d9bf7c445f99fc3630 (table rebalancer_extra_table_32 [id=dc79ef9727454a609aeb22c08a3349f3]) successfully deleted
I20240503 02:13:29.126295 17515 ts_tablet_manager.cc:1888] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:29.126529 17515 log.cc:1196] T d856c707315b47e996ee604a5ff66b31 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/d856c707315b47e996ee604a5ff66b31
I20240503 02:13:29.127707 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet d856c707315b47e996ee604a5ff66b31 (table rebalancer_extra_table_16 [id=3d07d59ab72149c9aa42399c6761daa5]) successfully deleted
I20240503 02:13:29.144600 17533 ts_tablet_manager.cc:902] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:29.165046 17533 tablet_copy_client.cc:308] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:29.165841 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet 5aa7ed62ebf44431a1a6db26ef73052a from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:59439)
I20240503 02:13:29.166054 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet 5aa7ed62ebf44431a1a6db26ef73052a from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:59439: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-5aa7ed62ebf44431a1a6db26ef73052a
I20240503 02:13:29.168309 10246 tablet_copy_source_session.cc:205] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:29.169957 17533 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5aa7ed62ebf44431a1a6db26ef73052a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:29.177721 17533 tablet_copy_client.cc:783] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:29.178045 17533 tablet_copy_client.cc:647] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:29.183032 17533 tablet_copy_client.cc:523] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:29.187481 17533 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:29.242884 17533 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:29.243577 17533 tablet_bootstrap.cc:492] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:29.244040 17533 ts_tablet_manager.cc:1367] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.057s user 0.036s sys 0.015s
I20240503 02:13:29.245234 17533 raft_consensus.cc:348] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:29.245541 17533 raft_consensus.cc:729] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:29.245774 17533 consensus_queue.cc:260] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:29.246507 17533 ts_tablet_manager.cc:1398] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.002s user 0.000s sys 0.004s
I20240503 02:13:29.256150 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-5aa7ed62ebf44431a1a6db26ef73052a received from {username='slave'} at 127.4.5.69:59439
I20240503 02:13:29.256345 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-5aa7ed62ebf44431a1a6db26ef73052a on tablet 5aa7ed62ebf44431a1a6db26ef73052a with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:29.327816 10487 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf"
dest_uuid: "c6ec2f1283994a3089741715f811c850"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:59692
I20240503 02:13:29.328181 10487 raft_consensus.cc:595] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:29.345784 17498 raft_consensus.cc:979] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: : Instructing follower 8ee3fd5ad3624ea1b9f681d24e9aabf2 to start an election
I20240503 02:13:29.346017 17498 raft_consensus.cc:1067] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Signalling peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 to start an election
I20240503 02:13:29.347831 13302 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf"
dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
from {username='slave'} at 127.4.5.68:44253
I20240503 02:13:29.348162 13302 raft_consensus.cc:482] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:29.348333 13302 raft_consensus.cc:3037] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:29.354010 13302 raft_consensus.cc:504] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:29.358047 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:29.358373 10487 raft_consensus.cc:3032] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:29.358462 10487 raft_consensus.cc:729] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:29.358953 10487 consensus_queue.cc:260] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:29.359337 13037 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
I20240503 02:13:29.359589 10487 raft_consensus.cc:3037] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:29.359681 13037 raft_consensus.cc:3037] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:29.363613 13037 raft_consensus.cc:2454] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 2.
I20240503 02:13:29.370527 10487 raft_consensus.cc:2454] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 2.
I20240503 02:13:29.372094 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f385792ed7c8465eb7cd64142778c9bf" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 3 } } ignore_live_leader: true dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:29.372387 10091 raft_consensus.cc:3037] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:29.375277 13302 leader_election.cc:288] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 election: Requested vote from peers 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:29.376065 10091 raft_consensus.cc:2454] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 2.
I20240503 02:13:29.387310 13245 leader_election.cc:302] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:13:29.388602 17444 raft_consensus.cc:2781] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:29.388846 17444 raft_consensus.cc:686] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:29.389292 17444 consensus_queue.cc:237] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 3, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:29.392979 17318 raft_consensus.cc:1050] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:29.393893 17318 consensus_queue.cc:237] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:29.397820 10490 raft_consensus.cc:1261] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:29.398057 13037 raft_consensus.cc:1261] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:29.398197 13302 raft_consensus.cc:1261] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:29.398545 17321 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:29.399183 17459 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:29.399637 17321 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:29.420526 17321 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:29.429246 9951 catalog_manager.cc:5484] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 1 to 2, leader changed from c6ec2f1283994a3089741715f811c850 (127.4.5.68) to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 2 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.441248 13040 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:29.442265 10490 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:29.453099 9944 catalog_manager.cc:5484] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:29.465924 13302 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:29.469889 13166 consensus_queue.cc:237] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:29.473120 10491 raft_consensus.cc:1261] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:29.473577 13040 raft_consensus.cc:1261] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:29.474341 17321 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.476120 17580 consensus_queue.cc:1035] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.478961 17321 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:29.479696 13040 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:29.480753 10491 raft_consensus.cc:2932] T 7afcfd9776ff4097a749ab8f91730d7b P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:29.484246 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 7afcfd9776ff4097a749ab8f91730d7b with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:29.489421 9947 catalog_manager.cc:5484] T 7afcfd9776ff4097a749ab8f91730d7b P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 3 to 4, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:29.495445 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 7afcfd9776ff4097a749ab8f91730d7b with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:29.498289 17589 tablet_replica.cc:330] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:29.498662 17589 raft_consensus.cc:2229] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:29.499091 17589 raft_consensus.cc:2258] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:29.501961 17589 ts_tablet_manager.cc:1875] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:29.510458 17589 ts_tablet_manager.cc:1888] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:29.510824 17589 log.cc:1196] T 7afcfd9776ff4097a749ab8f91730d7b P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/7afcfd9776ff4097a749ab8f91730d7b
I20240503 02:13:29.512269 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 7afcfd9776ff4097a749ab8f91730d7b (table rebalancer_extra_table_26 [id=88e38266de204766bdbdca073cb8c2b6]) successfully deleted
I20240503 02:13:29.527128 12906 raft_consensus.cc:1203] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:29.608167 10226 tablet_service.cc:1967] Received LeaderStepDown RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a"
dest_uuid: "eff8e8d11a4447218741bb994aa41082"
mode: GRACEFUL
from {username='slave'} at 127.0.0.1:57456
I20240503 02:13:29.608474 10226 raft_consensus.cc:595] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Received request to transfer leadership
I20240503 02:13:29.636543 17596 raft_consensus.cc:979] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: : Instructing follower 8d454a3607d042dc9559ed59c3ccd84b to start an election
I20240503 02:13:29.636750 17596 raft_consensus.cc:1067] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Signalling peer 8d454a3607d042dc9559ed59c3ccd84b to start an election
I20240503 02:13:29.637955 13040 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a"
dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b"
from {username='slave'} at 127.4.5.66:38795
I20240503 02:13:29.638257 13040 raft_consensus.cc:482] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting forced leader election (received explicit request)
I20240503 02:13:29.638402 13040 raft_consensus.cc:3037] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:29.640900 13040 raft_consensus.cc:504] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Starting forced leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:29.642315 10490 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:29.642342 10226 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "eff8e8d11a4447218741bb994aa41082"
I20240503 02:13:29.642585 10490 raft_consensus.cc:3037] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:29.642601 10226 raft_consensus.cc:3032] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Stepping down as leader of term 1
I20240503 02:13:29.642695 10226 raft_consensus.cc:729] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:29.643074 10226 consensus_queue.cc:260] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:29.643546 10226 raft_consensus.cc:3037] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:29.645263 10490 raft_consensus.cc:2454] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 2.
I20240503 02:13:29.645813 13040 leader_election.cc:288] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:29.646777 12906 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f8b006a679a4b8d8d3551310b78fb3a" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 2 candidate_status { last_received { term: 1 index: 8 } } ignore_live_leader: true dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:29.647061 12906 raft_consensus.cc:3037] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:29.647758 10226 raft_consensus.cc:2454] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 2.
I20240503 02:13:29.649152 12975 leader_election.cc:302] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 4 voters: 3 yes votes; 0 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, c6ec2f1283994a3089741715f811c850, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:29.649636 17513 raft_consensus.cc:2781] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:29.649806 17513 raft_consensus.cc:686] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 2 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:29.650249 17513 consensus_queue.cc:237] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:29.652014 12906 raft_consensus.cc:2454] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 2.
I20240503 02:13:29.655833 9951 catalog_manager.cc:5484] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 1 to 2, leader changed from eff8e8d11a4447218741bb994aa41082 (127.4.5.66) to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 2 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.662103 10089 consensus_queue.cc:237] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:29.665086 12906 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:29.665324 10029 consensus_peers.cc:485] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 49ce473013a942698081c389ace54b32. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.665777 10226 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:29.666299 17358 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:29.666647 17397 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:29.670500 17397 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:29.671900 12906 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:29.674335 10226 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:29.680107 9951 catalog_manager.cc:5484] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:29.701120 13302 consensus_queue.cc:237] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
W20240503 02:13:29.704224 13243 consensus_peers.cc:485] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8cf65124d3a4ccd8eded963fb7e92e8. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.704317 12906 raft_consensus.cc:1261] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.704594 13040 raft_consensus.cc:1261] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.705283 17650 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.705751 17444 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.711575 17444 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:29.712291 12906 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:29.722168 9943 catalog_manager.cc:5484] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:29.746778 13037 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:29.780393 10089 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:29.783778 17444 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.802811 13174 consensus_queue.cc:237] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:29.822405 13040 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:13:29.823177 13302 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:13:29.823968 13107 consensus_peers.cc:485] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0892c659fe1a43b796a73ee8b1783173. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.825213 17321 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.825661 17580 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.861426 17580 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:29.862571 13040 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:29.868158 9943 catalog_manager.cc:5484] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 4 to 5, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.871348 13302 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:29.897243 13040 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:29.898456 17650 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.919543 10089 consensus_queue.cc:237] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
W20240503 02:13:29.930565 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 818345467bd9400aa5fdd875b59f3249. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:29.932042 10491 raft_consensus.cc:1261] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.934762 13174 raft_consensus.cc:1261] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:29.937595 17358 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.939868 10491 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 2 index: 4. (index mismatch)
I20240503 02:13:29.942600 17444 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:29.943897 17397 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:29.945457 17397 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:29.946463 13305 consensus_queue.cc:237] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 2.4, Last appended by leader: 3, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:29.950232 13037 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:29.951038 17444 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:29.950918 10491 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:29.953989 16982 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 LEADER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:29.954684 13037 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:29.961433 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f385792ed7c8465eb7cd64142778c9bf with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:29.965276 9944 catalog_manager.cc:5484] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 5, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 2 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.968655 9951 catalog_manager.cc:5484] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:29.974546 10470 tablet_service.cc:1514] Processing DeleteTablet for tablet f385792ed7c8465eb7cd64142778c9bf with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 5) from {username='slave'} at 127.0.0.1:42260
I20240503 02:13:29.989683 13174 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:29.999836 17679 tablet_replica.cc:330] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:30.000121 17679 raft_consensus.cc:2229] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:30.000322 17679 raft_consensus.cc:2258] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:30.001998 17679 ts_tablet_manager.cc:1875] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:30.003420 10089 raft_consensus.cc:1261] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 2 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:30.004904 17444 consensus_queue.cc:1035] T f385792ed7c8465eb7cd64142778c9bf P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:13:30.007042 10091 raft_consensus.cc:2932] T f385792ed7c8465eb7cd64142778c9bf P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Committing config change with OpId 2.5: config changed from index 3 to 5, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:30.020313 17679 ts_tablet_manager.cc:1888] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.4
I20240503 02:13:30.020483 17679 log.cc:1196] T f385792ed7c8465eb7cd64142778c9bf P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/f385792ed7c8465eb7cd64142778c9bf
I20240503 02:13:30.021330 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet f385792ed7c8465eb7cd64142778c9bf (table rebalancer_dynamic_table [id=80fd16a658c24701a976069967996306]) successfully deleted
I20240503 02:13:30.033741 13305 consensus_queue.cc:237] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:30.037467 10490 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:30.038625 17513 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:30.052960 12906 raft_consensus.cc:1261] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.053681 17444 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:30.055467 13245 consensus_peers.cc:485] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: de58a8fe28a74c28a85c78ea8372ee0a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.059877 13037 raft_consensus.cc:1261] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.060714 17650 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.060583 16982 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:30.061388 12906 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:30.068262 13037 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:30.069463 9951 catalog_manager.cc:5484] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:30.081226 17691 ts_tablet_manager.cc:902] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:30.083901 17691 tablet_copy_client.cc:308] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.72:33671
I20240503 02:13:30.084834 13330 tablet_copy_service.cc:140] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Received BeginTabletCopySession request for tablet e8cf65124d3a4ccd8eded963fb7e92e8 from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:44253)
I20240503 02:13:30.085031 13330 tablet_copy_service.cc:161] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Beginning new tablet copy session on tablet e8cf65124d3a4ccd8eded963fb7e92e8 from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:44253: session id = c6ec2f1283994a3089741715f811c850-e8cf65124d3a4ccd8eded963fb7e92e8
I20240503 02:13:30.089179 13330 tablet_copy_source_session.cc:205] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.091410 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8cf65124d3a4ccd8eded963fb7e92e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.095089 10227 consensus_queue.cc:237] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:30.098819 17691 tablet_copy_client.cc:783] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.099107 17691 tablet_copy_client.cc:647] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.099988 13174 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:13:30.100152 10162 consensus_peers.cc:485] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ceb177d6952e4c978c8be93b5b026952. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.101037 10490 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.101254 17393 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.101500 17691 tablet_copy_client.cc:523] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.101624 17596 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.106763 17596 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:30.107694 10490 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:30.107936 13174 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:30.108846 17691 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:30.117810 17691 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.117386 9943 catalog_manager.cc:5484] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 6 to 7, NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } } }
I20240503 02:13:30.118238 17691 tablet_bootstrap.cc:492] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:30.118516 17691 ts_tablet_manager.cc:1367] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20240503 02:13:30.119691 17596 raft_consensus.cc:1050] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:30.120625 17596 consensus_queue.cc:237] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:30.129354 17691 raft_consensus.cc:348] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:30.129643 17691 raft_consensus.cc:729] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:30.131124 17691 consensus_queue.cc:260] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:30.131572 13166 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:30.132122 12906 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:30.132490 17691 ts_tablet_manager.cc:1398] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20240503 02:13:30.133086 17389 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:30.134395 10359 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:30.135212 17389 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:30.138324 17393 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:30.138656 13330 tablet_copy_service.cc:342] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-e8cf65124d3a4ccd8eded963fb7e92e8 received from {username='slave'} at 127.4.5.68:44253
I20240503 02:13:30.138947 13330 tablet_copy_service.cc:433] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: ending tablet copy session c6ec2f1283994a3089741715f811c850-e8cf65124d3a4ccd8eded963fb7e92e8 on tablet e8cf65124d3a4ccd8eded963fb7e92e8 with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:30.157860 10491 consensus_queue.cc:237] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:30.168998 17716 ts_tablet_manager.cc:902] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:30.173583 17716 tablet_copy_client.cc:308] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:30.174867 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 49ce473013a942698081c389ace54b32 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:60931)
I20240503 02:13:30.175087 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 49ce473013a942698081c389ace54b32 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:60931: session id = f93b103af083471b8e8c0376e9148aad-49ce473013a942698081c389ace54b32
I20240503 02:13:30.176337 13305 raft_consensus.cc:1261] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.177345 17498 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.178205 10115 tablet_copy_source_session.cc:205] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.182973 10227 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:30.184887 17313 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:13:30.186340 10427 consensus_peers.cc:485] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 274d302b083b42019a6f03305fe516e7. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.189371 10227 raft_consensus.cc:1261] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.192353 12906 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:30.192189 12908 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.193436 17418 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.195669 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 49ce473013a942698081c389ace54b32. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.206542 17418 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:30.209811 13302 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:30.210757 17716 tablet_copy_client.cc:783] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.211117 17716 tablet_copy_client.cc:647] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.210889 9951 catalog_manager.cc:5484] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.217458 17716 tablet_copy_client.cc:523] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.218715 17513 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:30.224663 17716 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:30.224691 17393 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.229528 9943 catalog_manager.cc:5484] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:30.240609 17691 ts_tablet_manager.cc:902] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:30.247344 13166 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.251387 10359 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.268427 13037 consensus_queue.cc:237] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 8, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:30.272724 10491 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:30.273769 17313 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:30.276598 17513 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.282359 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 9f8b006a679a4b8d8d3551310b78fb3a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:30.282316 10491 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
W20240503 02:13:30.286712 12842 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 51051.9us
I20240503 02:13:30.290989 9944 catalog_manager.cc:5484] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 8 to 10, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 2 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:30.302734 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f8b006a679a4b8d8d3551310b78fb3a with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 10) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:30.299676 10227 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:13:30.321249 12906 raft_consensus.cc:1261] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:30.323390 17313 consensus_queue.cc:1035] T 9f8b006a679a4b8d8d3551310b78fb3a P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:30.321147 10227 consensus_queue.cc:237] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:30.325726 17735 tablet_replica.cc:330] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:30.326048 17735 raft_consensus.cc:2229] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:30.326364 17735 pending_rounds.cc:62] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Trying to abort 1 pending ops.
I20240503 02:13:30.326496 17735 pending_rounds.cc:68] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Aborting op as it isn't in flight: id { term: 2 index: 9 } timestamp: 7023421069931544576 op_type: NO_OP noop_request { }
I20240503 02:13:30.326712 17735 raft_consensus.cc:2866] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20240503 02:13:30.326915 17735 raft_consensus.cc:2258] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:30.328403 17735 ts_tablet_manager.cc:1875] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:30.330014 13166 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:30.330921 17731 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:30.334213 12906 raft_consensus.cc:1261] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:30.335510 17711 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.337438 12908 raft_consensus.cc:2932] T 9f8b006a679a4b8d8d3551310b78fb3a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 8 to 10, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.340621 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5aa7ed62ebf44431a1a6db26ef73052a with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:30.346859 13174 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.355870 17596 consensus_queue.cc:1035] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:30.355944 9951 catalog_manager.cc:5484] T 5aa7ed62ebf44431a1a6db26ef73052a P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:30.362901 12908 raft_consensus.cc:2932] T 5aa7ed62ebf44431a1a6db26ef73052a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:30.365223 10338 tablet_service.cc:1514] Processing DeleteTablet for tablet 5aa7ed62ebf44431a1a6db26ef73052a with delete_type TABLET_DATA_TOMBSTONED (TS 47f901e617024baaa1865d12261ef84f not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42316
I20240503 02:13:30.379344 17736 ts_tablet_manager.cc:902] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:30.393723 17735 ts_tablet_manager.cc:1888] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 2.9
I20240503 02:13:30.393883 17735 log.cc:1196] T 9f8b006a679a4b8d8d3551310b78fb3a P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/9f8b006a679a4b8d8d3551310b78fb3a
I20240503 02:13:30.402068 17736 tablet_copy_client.cc:308] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:30.402115 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 9f8b006a679a4b8d8d3551310b78fb3a (table rebalance_test_table_5 [id=4f85ffd220044d0dae287adfc7633e88]) successfully deleted
I20240503 02:13:30.403074 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 818345467bd9400aa5fdd875b59f3249 from peer 47f901e617024baaa1865d12261ef84f ({username='slave'} at 127.4.5.67:60077)
I20240503 02:13:30.403296 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 818345467bd9400aa5fdd875b59f3249 from peer 47f901e617024baaa1865d12261ef84f at {username='slave'} at 127.4.5.67:60077: session id = 47f901e617024baaa1865d12261ef84f-818345467bd9400aa5fdd875b59f3249
I20240503 02:13:30.407003 10115 tablet_copy_source_session.cc:205] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.408746 17736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 818345467bd9400aa5fdd875b59f3249. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.412942 17691 tablet_copy_client.cc:308] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.71:46557
I20240503 02:13:30.413822 13195 tablet_copy_service.cc:140] P f93b103af083471b8e8c0376e9148aad: Received BeginTabletCopySession request for tablet 0892c659fe1a43b796a73ee8b1783173 from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:36981)
I20240503 02:13:30.414085 13195 tablet_copy_service.cc:161] P f93b103af083471b8e8c0376e9148aad: Beginning new tablet copy session on tablet 0892c659fe1a43b796a73ee8b1783173 from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:36981: session id = c6ec2f1283994a3089741715f811c850-0892c659fe1a43b796a73ee8b1783173
I20240503 02:13:30.416061 17736 tablet_copy_client.cc:783] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.416316 17736 tablet_copy_client.cc:647] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.420644 17739 tablet_replica.cc:330] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: stopping tablet replica
I20240503 02:13:30.420917 17739 raft_consensus.cc:2229] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:30.421182 17739 raft_consensus.cc:2258] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:30.422189 17739 ts_tablet_manager.cc:1875] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:30.423830 13195 tablet_copy_source_session.cc:205] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.425698 17691 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 0892c659fe1a43b796a73ee8b1783173. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.428728 17736 tablet_copy_client.cc:523] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.435055 17691 tablet_copy_client.cc:783] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.435457 17691 tablet_copy_client.cc:647] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.437162 17736 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: Bootstrap starting.
I20240503 02:13:30.437858 17691 tablet_copy_client.cc:523] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.442127 17691 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:30.442700 10091 consensus_queue.cc:237] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:30.457126 13037 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.457567 10359 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.458081 17448 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.459758 17691 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
W20240503 02:13:30.459944 10026 consensus_peers.cc:485] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5547b33324db46c886287b4ee5035e43. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.460151 17691 tablet_bootstrap.cc:492] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:30.460403 17691 ts_tablet_manager.cc:1367] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.004s
I20240503 02:13:30.461266 17691 raft_consensus.cc:348] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:30.461521 17691 raft_consensus.cc:729] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:30.461720 17691 consensus_queue.cc:260] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:30.463425 17358 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.467701 17691 ts_tablet_manager.cc:1398] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20240503 02:13:30.473182 17739 ts_tablet_manager.cc:1888] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:30.474819 13195 tablet_copy_service.cc:342] P f93b103af083471b8e8c0376e9148aad: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-0892c659fe1a43b796a73ee8b1783173 received from {username='slave'} at 127.4.5.68:36981
I20240503 02:13:30.475090 13195 tablet_copy_service.cc:433] P f93b103af083471b8e8c0376e9148aad: ending tablet copy session c6ec2f1283994a3089741715f811c850-0892c659fe1a43b796a73ee8b1783173 on tablet 0892c659fe1a43b796a73ee8b1783173 with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:30.477975 17739 log.cc:1196] T 5aa7ed62ebf44431a1a6db26ef73052a P 47f901e617024baaa1865d12261ef84f: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-2/wal/wals/5aa7ed62ebf44431a1a6db26ef73052a
I20240503 02:13:30.478385 17448 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:30.481639 9938 catalog_manager.cc:4830] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): tablet 5aa7ed62ebf44431a1a6db26ef73052a (table rebalance_test_table_13 [id=c8274877caff4abf9a02f6bbc4dec30c]) successfully deleted
I20240503 02:13:30.483672 17716 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=689 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.484308 10359 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:30.490661 9951 catalog_manager.cc:5484] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:30.492584 17736 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.492972 17736 tablet_bootstrap.cc:492] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: Bootstrap complete.
I20240503 02:13:30.493705 17736 ts_tablet_manager.cc:1367] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: Time spent bootstrapping tablet: real 0.057s user 0.012s sys 0.000s
I20240503 02:13:30.494400 17716 tablet_bootstrap.cc:492] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:30.494997 17716 ts_tablet_manager.cc:1367] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.270s user 0.061s sys 0.002s
I20240503 02:13:30.494943 13037 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:30.498858 17716 raft_consensus.cc:348] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:30.499233 17716 raft_consensus.cc:729] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:30.499583 17716 consensus_queue.cc:260] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:30.500655 17716 ts_tablet_manager.cc:1398] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20240503 02:13:30.501246 10491 raft_consensus.cc:1203] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.501756 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-49ce473013a942698081c389ace54b32 received from {username='slave'} at 127.4.5.71:60931
I20240503 02:13:30.501892 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session f93b103af083471b8e8c0376e9148aad-49ce473013a942698081c389ace54b32 on tablet 49ce473013a942698081c389ace54b32 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:30.507015 17736 raft_consensus.cc:348] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:30.507651 17736 raft_consensus.cc:729] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 47f901e617024baaa1865d12261ef84f, State: Initialized, Role: LEARNER
I20240503 02:13:30.507964 17736 consensus_queue.cc:260] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
I20240503 02:13:30.508953 17736 ts_tablet_manager.cc:1398] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f: Time spent starting tablet: real 0.015s user 0.000s sys 0.002s
I20240503 02:13:30.510246 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session 47f901e617024baaa1865d12261ef84f-818345467bd9400aa5fdd875b59f3249 received from {username='slave'} at 127.4.5.67:60077
I20240503 02:13:30.510381 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session 47f901e617024baaa1865d12261ef84f-818345467bd9400aa5fdd875b59f3249 on tablet 818345467bd9400aa5fdd875b59f3249 with peer 47f901e617024baaa1865d12261ef84f
I20240503 02:13:30.539060 13302 consensus_queue.cc:237] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:30.542773 10491 raft_consensus.cc:1261] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:30.543015 13245 consensus_peers.cc:485] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f38186393ba043a19f834d1d8780053a. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.543684 10227 raft_consensus.cc:1261] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.543747 17650 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.548837 17444 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.555110 17650 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:30.555470 10491 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:30.559829 10227 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:30.562439 9951 catalog_manager.cc:5484] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index -1 to 2, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:13:30.589916 13302 consensus_queue.cc:237] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } }
W20240503 02:13:30.593753 13245 consensus_peers.cc:485] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af66d6e994ab4049964f4218e05ef886. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.595935 13037 raft_consensus.cc:1261] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.595986 13174 raft_consensus.cc:1261] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:13:30.596741 17650 consensus_queue.cc:1035] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.599555 16982 consensus_queue.cc:1035] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:30.601856 17444 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:30.602607 13037 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:30.609107 9944 catalog_manager.cc:5484] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:30.611492 17716 ts_tablet_manager.cc:902] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:30.611666 13174 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: NON_VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: true } } }
I20240503 02:13:30.657697 17716 tablet_copy_client.cc:308] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.68:35025
I20240503 02:13:30.658761 10511 tablet_copy_service.cc:140] P c6ec2f1283994a3089741715f811c850: Received BeginTabletCopySession request for tablet 274d302b083b42019a6f03305fe516e7 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:33045)
I20240503 02:13:30.658955 10511 tablet_copy_service.cc:161] P c6ec2f1283994a3089741715f811c850: Beginning new tablet copy session on tablet 274d302b083b42019a6f03305fe516e7 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:33045: session id = f93b103af083471b8e8c0376e9148aad-274d302b083b42019a6f03305fe516e7
I20240503 02:13:30.662817 10511 tablet_copy_source_session.cc:205] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.667769 17780 ts_tablet_manager.cc:902] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:30.668557 17716 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 274d302b083b42019a6f03305fe516e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.678680 17716 tablet_copy_client.cc:783] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.679118 17716 tablet_copy_client.cc:647] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.683878 10359 consensus_queue.cc:237] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:30.684386 17716 tablet_copy_client.cc:523] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.688796 17716 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: Bootstrap starting.
I20240503 02:13:30.691051 10227 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.691735 17437 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.693596 17780 tablet_copy_client.cc:308] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.72:33671
I20240503 02:13:30.694557 13330 tablet_copy_service.cc:140] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Received BeginTabletCopySession request for tablet de58a8fe28a74c28a85c78ea8372ee0a from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:54353)
I20240503 02:13:30.694870 13330 tablet_copy_service.cc:161] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Beginning new tablet copy session on tablet de58a8fe28a74c28a85c78ea8372ee0a from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:54353: session id = 031a0afb01b6469299fa829fa3467ef7-de58a8fe28a74c28a85c78ea8372ee0a
W20240503 02:13:30.696193 10292 consensus_peers.cc:485] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f -> Peer e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): Couldn't send request to peer e3cbe34992dc4b77b9aa9e9a4c143e78. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35aea508f59f4348b05d93a5afdde4ba. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:30.696779 13302 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 47f901e617024baaa1865d12261ef84f: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:30.699702 17781 ts_tablet_manager.cc:902] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:30.699672 13330 tablet_copy_source_session.cc:205] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.706799 13174 raft_consensus.cc:1203] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:30.716845 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de58a8fe28a74c28a85c78ea8372ee0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.725813 17437 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:30.735801 17439 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:30.739569 13302 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:30.740610 10226 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:30.752015 9943 catalog_manager.cc:5484] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 6 to 7, NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) added. New cstate: current_term: 1 leader_uuid: "47f901e617024baaa1865d12261ef84f" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } } }
I20240503 02:13:30.763590 17780 tablet_copy_client.cc:783] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.764019 17780 tablet_copy_client.cc:647] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.766800 17780 tablet_copy_client.cc:523] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.769930 17716 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.771216 17781 tablet_copy_client.cc:308] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.66:42397
I20240503 02:13:30.771832 17716 tablet_bootstrap.cc:492] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: Bootstrap complete.
I20240503 02:13:30.772229 17716 ts_tablet_manager.cc:1367] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: Time spent bootstrapping tablet: real 0.083s user 0.011s sys 0.033s
I20240503 02:13:30.772595 10246 tablet_copy_service.cc:140] P eff8e8d11a4447218741bb994aa41082: Received BeginTabletCopySession request for tablet ceb177d6952e4c978c8be93b5b026952 from peer 8d454a3607d042dc9559ed59c3ccd84b ({username='slave'} at 127.4.5.70:38111)
I20240503 02:13:30.772769 10246 tablet_copy_service.cc:161] P eff8e8d11a4447218741bb994aa41082: Beginning new tablet copy session on tablet ceb177d6952e4c978c8be93b5b026952 from peer 8d454a3607d042dc9559ed59c3ccd84b at {username='slave'} at 127.4.5.70:38111: session id = 8d454a3607d042dc9559ed59c3ccd84b-ceb177d6952e4c978c8be93b5b026952
I20240503 02:13:30.773803 17780 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:30.774972 17716 raft_consensus.cc:348] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:30.775280 17716 raft_consensus.cc:729] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Initialized, Role: LEARNER
I20240503 02:13:30.775475 17716 consensus_queue.cc:260] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:13:30.775696 10246 tablet_copy_source_session.cc:205] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:30.777452 17716 ts_tablet_manager.cc:1398] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20240503 02:13:30.778228 10511 tablet_copy_service.cc:342] P c6ec2f1283994a3089741715f811c850: Request end of tablet copy session f93b103af083471b8e8c0376e9148aad-274d302b083b42019a6f03305fe516e7 received from {username='slave'} at 127.4.5.71:33045
I20240503 02:13:30.778359 10511 tablet_copy_service.cc:433] P c6ec2f1283994a3089741715f811c850: ending tablet copy session f93b103af083471b8e8c0376e9148aad-274d302b083b42019a6f03305fe516e7 on tablet 274d302b083b42019a6f03305fe516e7 with peer f93b103af083471b8e8c0376e9148aad
I20240503 02:13:30.779035 17781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ceb177d6952e4c978c8be93b5b026952. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:30.784932 10359 raft_consensus.cc:1203] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:30.819185 17781 tablet_copy_client.cc:783] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:30.819542 17781 tablet_copy_client.cc:647] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:30.836582 17781 tablet_copy_client.cc:523] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:30.841487 10491 raft_consensus.cc:1203] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20240503 02:13:30.848284 17780 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.849035 17780 tablet_bootstrap.cc:492] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:30.849455 17780 ts_tablet_manager.cc:1367] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.076s user 0.005s sys 0.006s
I20240503 02:13:30.850188 17781 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap starting.
I20240503 02:13:30.850769 17780 raft_consensus.cc:348] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:30.851179 17780 raft_consensus.cc:729] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:30.851465 17780 consensus_queue.cc:260] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:30.852491 17780 ts_tablet_manager.cc:1398] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20240503 02:13:30.865011 13330 tablet_copy_service.cc:342] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-de58a8fe28a74c28a85c78ea8372ee0a received from {username='slave'} at 127.4.5.65:54353
I20240503 02:13:30.865257 13330 tablet_copy_service.cc:433] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-de58a8fe28a74c28a85c78ea8372ee0a on tablet de58a8fe28a74c28a85c78ea8372ee0a with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:30.949007 17780 ts_tablet_manager.cc:902] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:30.989501 17781 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:30.990736 17781 tablet_bootstrap.cc:492] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: Bootstrap complete.
I20240503 02:13:30.991106 17781 ts_tablet_manager.cc:1367] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent bootstrapping tablet: real 0.141s user 0.054s sys 0.010s
I20240503 02:13:30.992156 17781 raft_consensus.cc:348] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:30.992529 17781 raft_consensus.cc:729] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Initialized, Role: LEARNER
I20240503 02:13:30.992805 17781 consensus_queue.cc:260] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:13:30.995443 17780 tablet_copy_client.cc:308] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.72:33671
I20240503 02:13:30.996241 17781 ts_tablet_manager.cc:1398] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20240503 02:13:30.997205 10246 tablet_copy_service.cc:342] P eff8e8d11a4447218741bb994aa41082: Request end of tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-ceb177d6952e4c978c8be93b5b026952 received from {username='slave'} at 127.4.5.70:38111
I20240503 02:13:30.997331 10246 tablet_copy_service.cc:433] P eff8e8d11a4447218741bb994aa41082: ending tablet copy session 8d454a3607d042dc9559ed59c3ccd84b-ceb177d6952e4c978c8be93b5b026952 on tablet ceb177d6952e4c978c8be93b5b026952 with peer 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:30.999454 13330 tablet_copy_service.cc:140] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Received BeginTabletCopySession request for tablet f38186393ba043a19f834d1d8780053a from peer 031a0afb01b6469299fa829fa3467ef7 ({username='slave'} at 127.4.5.65:54353)
I20240503 02:13:30.999662 13330 tablet_copy_service.cc:161] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Beginning new tablet copy session on tablet f38186393ba043a19f834d1d8780053a from peer 031a0afb01b6469299fa829fa3467ef7 at {username='slave'} at 127.4.5.65:54353: session id = 031a0afb01b6469299fa829fa3467ef7-f38186393ba043a19f834d1d8780053a
I20240503 02:13:31.000099 17812 ts_tablet_manager.cc:902] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:31.003134 13330 tablet_copy_source_session.cc:205] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:31.005676 17812 tablet_copy_client.cc:308] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:31.006819 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 5547b33324db46c886287b4ee5035e43 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:56811)
I20240503 02:13:31.006997 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 5547b33324db46c886287b4ee5035e43 from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:56811: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-5547b33324db46c886287b4ee5035e43
I20240503 02:13:31.007367 17780 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f38186393ba043a19f834d1d8780053a. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.010284 10115 tablet_copy_source_session.cc:205] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:31.013901 17812 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5547b33324db46c886287b4ee5035e43. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:31.015496 17780 tablet_copy_client.cc:783] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:31.015758 17780 tablet_copy_client.cc:647] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:31.017527 17780 tablet_copy_client.cc:523] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:31.022087 17812 tablet_copy_client.cc:783] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:31.022503 17812 tablet_copy_client.cc:647] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:31.024761 17780 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap starting.
I20240503 02:13:31.025197 17812 tablet_copy_client.cc:523] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:31.030153 17812 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:31.036389 17771 raft_consensus.cc:1050] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:31.037496 17771 consensus_queue.cc:237] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:31.041530 17780 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:31.041966 17780 tablet_bootstrap.cc:492] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: Bootstrap complete.
I20240503 02:13:31.042263 17780 ts_tablet_manager.cc:1367] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.002s
I20240503 02:13:31.042301 10491 raft_consensus.cc:1261] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.043071 17780 raft_consensus.cc:348] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:31.043356 17780 raft_consensus.cc:729] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Initialized, Role: LEARNER
I20240503 02:13:31.043272 17444 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.043550 17780 consensus_queue.cc:260] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:13:31.044183 17780 ts_tablet_manager.cc:1398] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20240503 02:13:31.048640 13040 raft_consensus.cc:1261] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.049000 12908 raft_consensus.cc:1261] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.049976 17444 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.050706 17444 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.084198 13037 raft_consensus.cc:1203] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:31.087839 13330 tablet_copy_service.cc:342] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Request end of tablet copy session 031a0afb01b6469299fa829fa3467ef7-f38186393ba043a19f834d1d8780053a received from {username='slave'} at 127.4.5.65:54353
I20240503 02:13:31.088085 13330 tablet_copy_service.cc:433] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: ending tablet copy session 031a0afb01b6469299fa829fa3467ef7-f38186393ba043a19f834d1d8780053a on tablet f38186393ba043a19f834d1d8780053a with peer 031a0afb01b6469299fa829fa3467ef7
I20240503 02:13:31.106261 17771 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.111827 12906 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.115324 10491 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.125289 9944 catalog_manager.cc:5484] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.162724 13302 consensus_queue.cc:237] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:31.165920 17833 ts_tablet_manager.cc:902] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: Initiating tablet copy from peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:31.172297 17812 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=635 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:31.173264 17812 tablet_bootstrap.cc:492] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:31.173663 17812 ts_tablet_manager.cc:1367] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.144s user 0.054s sys 0.005s
I20240503 02:13:31.174790 17812 raft_consensus.cc:348] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:31.175287 17812 raft_consensus.cc:729] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:31.175654 17812 consensus_queue.cc:260] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:31.176983 17812 ts_tablet_manager.cc:1398] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20240503 02:13:31.177774 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-5547b33324db46c886287b4ee5035e43 received from {username='slave'} at 127.4.5.69:56811
I20240503 02:13:31.177901 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-5547b33324db46c886287b4ee5035e43 on tablet 5547b33324db46c886287b4ee5035e43 with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:31.178296 10491 raft_consensus.cc:1261] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:31.179517 17650 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:31.183102 10089 raft_consensus.cc:1203] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:31.187326 17771 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.195174 10491 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.196836 17814 raft_consensus.cc:1050] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:31.196986 12903 raft_consensus.cc:1261] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:31.197731 17650 consensus_queue.cc:1035] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:31.198061 17814 consensus_queue.cc:237] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:31.199977 12903 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.209925 13166 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.211072 17397 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.213297 12908 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.214280 17397 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.216352 10226 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.223286 17397 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.231321 13174 raft_consensus.cc:1203] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:31.232221 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet e8cf65124d3a4ccd8eded963fb7e92e8 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.237051 17358 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:31.238958 9951 catalog_manager.cc:5484] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) evicted. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:31.239037 13040 raft_consensus.cc:2932] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.242014 17835 ts_tablet_manager.cc:902] T af66d6e994ab4049964f4218e05ef886 P 47f901e617024baaa1865d12261ef84f: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:31.244767 10226 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
F20240503 02:13:31.242436 17835 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-09468c67719244c18d200463892a4f84.queue_time_us, tablet-download-pool-09468c67719244c18d200463892a4f84.queue_time_us) (tablet-download-pool-09468c67719244c18d200463892a4f84.run_cpu_time_us, tablet-download-pool-09468c67719244c18d200463892a4f84.run_cpu_time_us) (tablet-download-pool-09468c67719244c18d200463892a4f84.run_wall_time_us, tablet-download-pool-09468c67719244c18d200463892a4f84.run_wall_time_us) (tablet-download-pool-09a142900b5c4909ad16b4abc70ee952.queue_time_us, tablet-download-pool-09a142900b5c4909ad16b4abc70ee952.queue_time_us) (tablet-download-pool-09a142900b5c4909ad16b4abc70ee952.run_cpu_time_us, tablet-download-pool-09a142900b5c4909ad16b4abc70ee952.run_cpu_time_us) (tablet-download-pool-09a142900b5c4909ad16b4abc70ee952.run_wall_time_us, tablet-download-pool-09a142900b5c4909ad16b4abc70ee952.run_wall_time_us) (tablet-download-pool-09e0373a72654a1082464fac8d428483.queue_time_us, tablet-download-pool-09e0373a72654a1082464fac8d428483.queue_time_us) (tablet-download-pool-09e0373a72654a1082464fac8d428483.run_cpu_time_us, tablet-download-pool-09e0373a72654a1082464fac8d428483.run_cpu_time_us) (tablet-download-pool-09e0373a72654a1082464fac8d428483.run_wall_time_us, tablet-download-pool-09e0373a72654a1082464fac8d428483.run_wall_time_us) (tablet-download-pool-0bee269120a44b53b5bacc722f6742a9.queue_time_us, tablet-download-pool-0bee269120a44b53b5bacc722f6742a9.queue_time_us) (tablet-download-pool-0bee269120a44b53b5bacc722f6742a9.run_cpu_time_us, tablet-download-pool-0bee269120a44b53b5bacc722f6742a9.run_cpu_time_us) (tablet-download-pool-0bee269120a44b53b5bacc722f6742a9.run_wall_time_us, tablet-download-pool-0bee269120a44b53b5bacc722f6742a9.run_wall_time_us) (tablet-download-pool-317ce62718cb412bbf04569cab14309f.queue_time_us, tablet-download-pool-317ce62718cb412bbf04569cab14309f.queue_time_us) (tablet-download-pool-317ce62718cb412bbf04569cab14309f.run_cpu_time_us, tablet-download-pool-317ce62718cb412bbf04569cab14309f.run_cpu_time_us) (tablet-download-pool-317ce62718cb412bbf04569cab14309f.run_wall_time_us, tablet-download-pool-317ce62718cb412bbf04569cab14309f.run_wall_time_us) (tablet-download-pool-46e5690a42d44500b3e6ca24d771affc.queue_time_us, tablet-download-pool-46e5690a42d44500b3e6ca24d771affc.queue_time_us) (tablet-download-pool-46e5690a42d44500b3e6ca24d771affc.run_cpu_time_us, tablet-download-pool-46e5690a42d44500b3e6ca24d771affc.run_cpu_time_us) (tablet-download-pool-46e5690a42d44500b3e6ca24d771affc.run_wall_time_us, tablet-download-pool-46e5690a42d44500b3e6ca24d771affc.run_wall_time_us) (tablet-download-pool-4c014b2069ae42788aae87d8304fa155.queue_time_us, tablet-download-pool-4c014b2069ae42788aae87d8304fa155.queue_time_us) (tablet-download-pool-4c014b2069ae42788aae87d8304fa155.run_cpu_time_us, tablet-download-pool-4c014b2069ae42788aae87d8304fa155.run_cpu_time_us) (tablet-download-pool-4c014b2069ae42788aae87d8304fa155.run_wall_time_us, tablet-download-pool-4c014b2069ae42788aae87d8304fa155.run_wall_time_us) (tablet-download-pool-67be061441154fbba1e227d32474cd5e.queue_time_us, tablet-download-pool-67be061441154fbba1e227d32474cd5e.queue_time_us) (tablet-download-pool-67be061441154fbba1e227d32474cd5e.run_cpu_time_us, tablet-download-pool-67be061441154fbba1e227d32474cd5e.run_cpu_time_us) (tablet-download-pool-67be061441154fbba1e227d32474cd5e.run_wall_time_us, tablet-download-pool-67be061441154fbba1e227d32474cd5e.run_wall_time_us) (tablet-download-pool-818345467bd9400aa5fdd875b59f3249.queue_time_us, tablet-download-pool-818345467bd9400aa5fdd875b59f3249.queue_time_us) (tablet-download-pool-818345467bd9400aa5fdd875b59f3249.run_cpu_time_us, tablet-download-pool-818345467bd9400aa5fdd875b59f3249.run_cpu_time_us) (tablet-download-pool-818345467bd9400aa5fdd875b59f3249.run_wall_time_us, tablet-download-pool-818345467bd9400aa5fdd875b59f3249.run_wall_time_us) (tablet-download-pool-9abc528ca83640c3bf66022cf2c9b914.queue_time_us, tablet-download-pool-9abc528ca83640c3bf66022cf2c9b914.queue_time_us) (tablet-download-pool-9abc528ca83640c3bf66022cf2c9b914.run_cpu_time_us, tablet-download-pool-9abc528ca83640c3bf66022cf2c9b914.run_cpu_time_us) (tablet-download-pool-9abc528ca83640c3bf66022cf2c9b914.run_wall_time_us, tablet-download-pool-9abc528ca83640c3bf66022cf2c9b914.run_wall_time_us) (tablet-download-pool-ac9acb6fb9c940ce83922596c849c4d8.queue_time_us, tablet-download-pool-ac9acb6fb9c940ce83922596c849c4d8.queue_time_us) (tablet-download-pool-ac9acb6fb9c940ce83922596c849c4d8.run_cpu_time_us, tablet-download-pool-ac9acb6fb9c940ce83922596c849c4d8.run_cpu_time_us) (tablet-download-pool-ac9acb6fb9c940ce83922596c849c4d8.run_wall_time_us, tablet-download-pool-ac9acb6fb9c940ce83922596c849c4d8.run_wall_time_us) (tablet-download-pool-af66d6e994ab4049964f4218e05ef886.queue_time_us, tablet-download-pool-af66d6e994ab4049964f4218e05ef886.queue_time_us) (tablet-download-pool-c2a328e790e844b5aa9a1cf0fa16ee8c.queue_time_us, tablet-download-pool-c2a328e790e844b5aa9a1cf0fa16ee8c.queue_time_us) (tablet-download-pool-c2a328e790e844b5aa9a1cf0fa16ee8c.run_cpu_time_us, tablet-download-pool-c2a328e790e844b5aa9a1cf0fa16ee8c.run_cpu_time_us) (tablet-download-pool-c2a328e790e844b5aa9a1cf0fa16ee8c.run_wall_time_us, tablet-download-pool-c2a328e790e844b5aa9a1cf0fa16ee8c.run_wall_time_us) (tablet-download-pool-db2da39193164d1ebf830282258d47d7.queue_time_us, tablet-download-pool-db2da39193164d1ebf830282258d47d7.queue_time_us) (tablet-download-pool-db2da39193164d1ebf830282258d47d7.run_cpu_time_us, tablet-download-pool-db2da39193164d1ebf830282258d47d7.run_cpu_time_us) (tablet-download-pool-db2da39193164d1ebf830282258d47d7.run_wall_time_us, tablet-download-pool-db2da39193164d1ebf830282258d47d7.run_wall_time_us) (tablet-download-pool-f1f1661a75a74746a4485b88b8e5b185.queue_time_us, tablet-download-pool-f1f1661a75a74746a4485b88b8e5b185.queue_time_us) (tablet-download-pool-f1f1661a75a74746a4485b88b8e5b185.run_cpu_time_us, tablet-download-pool-f1f1661a75a74746a4485b88b8e5b185.run_cpu_time_us) (tablet-download-pool-f1f1661a75a74746a4485b88b8e5b185.run_wall_time_us, tablet-download-pool-f1f1661a75a74746a4485b88b8e5b185.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
*** Check failure stack trace: ***
I20240503 02:13:31.251879 17448 raft_consensus.cc:1050] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER 47f901e617024baaa1865d12261ef84f to VOTER
I20240503 02:13:31.253180 17448 consensus_queue.cc:237] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
I20240503 02:13:31.254963 13019 tablet_service.cc:1514] Processing DeleteTablet for tablet e8cf65124d3a4ccd8eded963fb7e92e8 with delete_type TABLET_DATA_TOMBSTONED (TS 8d454a3607d042dc9559ed59c3ccd84b not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:52066
@ 0x7fb4402ebdcd google::LogMessage::Fail() at ??:0
I20240503 02:13:31.269428 13166 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
@ 0x7fb4402efb93 google::LogMessage::SendToLog() at ??:0
I20240503 02:13:31.289876 9944 catalog_manager.cc:5484] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:31.299389 12908 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:31.323367 17833 tablet_copy_client.cc:308] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.67:41899
I20240503 02:13:31.323474 10091 consensus_queue.cc:237] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:31.325752 13166 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.326185 12908 raft_consensus.cc:1261] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.327050 10380 tablet_copy_service.cc:140] P 47f901e617024baaa1865d12261ef84f: Received BeginTabletCopySession request for tablet 35aea508f59f4348b05d93a5afdde4ba from peer e3cbe34992dc4b77b9aa9e9a4c143e78 ({username='slave'} at 127.4.5.69:56263)
I20240503 02:13:31.327009 17846 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.327236 10380 tablet_copy_service.cc:161] P 47f901e617024baaa1865d12261ef84f: Beginning new tablet copy session on tablet 35aea508f59f4348b05d93a5afdde4ba from peer e3cbe34992dc4b77b9aa9e9a4c143e78 at {username='slave'} at 127.4.5.69:56263: session id = e3cbe34992dc4b77b9aa9e9a4c143e78-35aea508f59f4348b05d93a5afdde4ba
I20240503 02:13:31.329860 17814 consensus_queue.cc:1035] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.330394 10380 tablet_copy_source_session.cc:205] T 35aea508f59f4348b05d93a5afdde4ba P 47f901e617024baaa1865d12261ef84f: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:31.333740 17833 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35aea508f59f4348b05d93a5afdde4ba. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fb4402eb7cc google::LogMessage::Flush() at ??:0
I20240503 02:13:31.335783 17846 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
@ 0x7fb4402ecf59 google::LogMessageFatal::~LogMessageFatal() at ??:0
I20240503 02:13:31.340983 12908 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:31.341564 13174 raft_consensus.cc:2932] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:31.342741 17833 tablet_copy_client.cc:783] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:31.344753 10359 raft_consensus.cc:1261] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.345422 17358 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.345623 10491 raft_consensus.cc:1261] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.347424 13172 raft_consensus.cc:1261] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.352773 17844 tablet_replica.cc:330] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: stopping tablet replica
I20240503 02:13:31.353405 9944 catalog_manager.cc:5484] T 49ce473013a942698081c389ace54b32 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 8 to 9, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:31.353085 17844 raft_consensus.cc:2229] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.356282 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 49ce473013a942698081c389ace54b32 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.346174 17397 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.013s
I20240503 02:13:31.359103 17833 tablet_copy_client.cc:647] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:31.364316 17844 raft_consensus.cc:2258] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.365672 17844 ts_tablet_manager.cc:1875] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.371268 17833 tablet_copy_client.cc:523] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20240503 02:13:31.371907 17358 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
@ 0x7fb44302c742 kudu::TraceMetrics::InternName() at ??:0
I20240503 02:13:31.386729 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 49ce473013a942698081c389ace54b32 with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:31.391322 17814 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:31.396426 10358 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:31.399070 10490 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:31.407049 9944 catalog_manager.cc:5484] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f reported cstate change: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:31.409911 13166 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 47f901e617024baaa1865d12261ef84f (127.4.5.67) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:31.412318 17807 raft_consensus.cc:1050] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
@ 0x7fb442ff8f8b kudu::ThreadPool::ThreadPool() at ??:0
I20240503 02:13:31.414223 17807 consensus_queue.cc:237] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:31.417719 17844 ts_tablet_manager.cc:1888] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:31.417910 17844 log.cc:1196] T e8cf65124d3a4ccd8eded963fb7e92e8 P 8d454a3607d042dc9559ed59c3ccd84b: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-5/wal/wals/e8cf65124d3a4ccd8eded963fb7e92e8
I20240503 02:13:31.419382 10091 consensus_queue.cc:237] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } }
@ 0x7fb442ff2cd6 kudu::ThreadPoolBuilder::Build() at ??:0
I20240503 02:13:31.422595 10359 raft_consensus.cc:1261] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:31.423619 17358 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:31.427039 9939 catalog_manager.cc:4830] TS 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): tablet e8cf65124d3a4ccd8eded963fb7e92e8 (table rebalancer_extra_table_28 [id=c8e9200b4bb041f2b852f533100bfa61]) successfully deleted
I20240503 02:13:31.432065 10490 raft_consensus.cc:1261] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:31.432917 17814 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:31.435705 17846 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:31.441444 17833 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap starting.
I20240503 02:13:31.444387 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 818345467bd9400aa5fdd875b59f3249 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.446985 17855 tablet_replica.cc:330] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:31.446959 10359 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P 47f901e617024baaa1865d12261ef84f [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
I20240503 02:13:31.447263 17855 raft_consensus.cc:2229] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.450374 17855 raft_consensus.cc:2258] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.451206 9943 catalog_manager.cc:5484] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:31.451982 17855 ts_tablet_manager.cc:1875] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.458294 13153 tablet_service.cc:1514] Processing DeleteTablet for tablet 818345467bd9400aa5fdd875b59f3249 with delete_type TABLET_DATA_TOMBSTONED (TS f93b103af083471b8e8c0376e9148aad not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:60990
I20240503 02:13:31.460623 10490 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:31.461483 13040 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:31.461385 17803 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:31.462211 17803 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:31.462303 10491 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } }
@ 0x7fb4524468de kudu::tserver::TabletCopyClient::TabletCopyClient() at ??:0
I20240503 02:13:31.482254 12903 raft_consensus.cc:1203] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20240503 02:13:31.482915 13296 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20240503 02:13:31.483731 17859 tablet_replica.cc:330] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: stopping tablet replica
I20240503 02:13:31.484042 17859 raft_consensus.cc:2229] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.483942 17803 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20240503 02:13:31.484378 17859 raft_consensus.cc:2258] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:31.485219 10091 raft_consensus.cc:1203] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:31.488759 17859 ts_tablet_manager.cc:1875] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.508975 17859 ts_tablet_manager.cc:1888] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:31.509218 17859 log.cc:1196] T 818345467bd9400aa5fdd875b59f3249 P f93b103af083471b8e8c0376e9148aad: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-6/wal/wals/818345467bd9400aa5fdd875b59f3249
I20240503 02:13:31.510485 9939 catalog_manager.cc:4830] TS f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): tablet 818345467bd9400aa5fdd875b59f3249 (table rebalancer_extra_table_38 [id=ac7d3e3f03cc445382c1695f49facde9]) successfully deleted
@ 0x7fb45245edaf kudu::tserver::RemoteTabletCopyClient::RemoteTabletCopyClient() at ??:0
I20240503 02:13:31.523618 17803 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.536271 10487 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.540719 17596 raft_consensus.cc:1050] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER 8d454a3607d042dc9559ed59c3ccd84b to VOTER
I20240503 02:13:31.541769 17596 consensus_queue.cc:237] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:31.546727 13302 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.547729 9944 catalog_manager.cc:5484] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.552537 13040 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.557205 17855 ts_tablet_manager.cc:1888] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:31.557356 17855 log.cc:1196] T 49ce473013a942698081c389ace54b32 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/49ce473013a942698081c389ace54b32
I20240503 02:13:31.563917 13036 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.564575 13166 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.566219 10487 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:31.566771 17393 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.567111 17872 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.567765 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 49ce473013a942698081c389ace54b32 (table rebalance_test_table_19 [id=fad6736bc30c4b9f8300ba014bad4951]) successfully deleted
I20240503 02:13:31.568053 17871 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:31.580157 13174 consensus_queue.cc:237] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:31.584491 13036 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:31.584834 10490 raft_consensus.cc:1261] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:13:31.585518 17860 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:31.588654 17803 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
@ 0x7fb452589480 kudu::tserver::TSTabletManager::RunTabletCopy() at ??:0
I20240503 02:13:31.593822 13036 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.600082 9939 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 0892c659fe1a43b796a73ee8b1783173 with cas_config_opid_index 6: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.602967 17807 consensus_queue.cc:1035] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:31.602924 17871 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:31.605317 10490 raft_consensus.cc:2932] T 0892c659fe1a43b796a73ee8b1783173 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:31.604836 9944 catalog_manager.cc:5484] T 0892c659fe1a43b796a73ee8b1783173 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 6 to 7, VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) evicted. New cstate: current_term: 1 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:31.610970 13290 tablet_service.cc:1514] Processing DeleteTablet for tablet 0892c659fe1a43b796a73ee8b1783173 with delete_type TABLET_DATA_TOMBSTONED (TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 not found in new config with opid_index 7) from {username='slave'} at 127.0.0.1:42238
I20240503 02:13:31.618041 10491 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:31.624061 9947 catalog_manager.cc:5484] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:31.621551 13036 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:31.637084 17879 tablet_replica.cc:330] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: stopping tablet replica
I20240503 02:13:31.638469 17879 raft_consensus.cc:2229] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.638900 17879 raft_consensus.cc:2258] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fb4525c1a8d kudu::tserver::TabletCopyRunnable::Run() at ??:0
I20240503 02:13:31.642324 10226 consensus_queue.cc:237] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } }
I20240503 02:13:31.645856 17879 ts_tablet_manager.cc:1875] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:31.654943 13036 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.654973 13166 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:31.661623 17393 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.662259 17879 ts_tablet_manager.cc:1888] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.6
I20240503 02:13:31.662427 17879 log.cc:1196] T 0892c659fe1a43b796a73ee8b1783173 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-7/wal/wals/0892c659fe1a43b796a73ee8b1783173
I20240503 02:13:31.663458 13166 raft_consensus.cc:1261] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:31.664132 17872 consensus_queue.cc:1035] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:31.666715 17393 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:31.668123 13166 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:31.674453 9936 catalog_manager.cc:4830] TS 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): tablet 0892c659fe1a43b796a73ee8b1783173 (table rebalancer_extra_table_12 [id=8fba9d01a28f4cbfa8da10e9c1c63353]) successfully deleted
I20240503 02:13:31.676040 17833 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=676 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:31.676560 17833 tablet_bootstrap.cc:492] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: Bootstrap complete.
I20240503 02:13:31.676867 17833 ts_tablet_manager.cc:1367] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent bootstrapping tablet: real 0.239s user 0.049s sys 0.009s
I20240503 02:13:31.677837 17833 raft_consensus.cc:348] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:31.678170 17833 raft_consensus.cc:729] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e3cbe34992dc4b77b9aa9e9a4c143e78, State: Initialized, Role: LEARNER
I20240503 02:13:31.678376 17833 consensus_queue.cc:260] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:31.684329 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet ceb177d6952e4c978c8be93b5b026952 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.685209 17833 ts_tablet_manager.cc:1398] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20240503 02:13:31.686142 10380 tablet_copy_service.cc:342] P 47f901e617024baaa1865d12261ef84f: Request end of tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-35aea508f59f4348b05d93a5afdde4ba received from {username='slave'} at 127.4.5.69:56263
I20240503 02:13:31.686275 10380 tablet_copy_service.cc:433] P 47f901e617024baaa1865d12261ef84f: ending tablet copy session e3cbe34992dc4b77b9aa9e9a4c143e78-35aea508f59f4348b05d93a5afdde4ba on tablet 35aea508f59f4348b05d93a5afdde4ba with peer e3cbe34992dc4b77b9aa9e9a4c143e78
I20240503 02:13:31.686019 9947 catalog_manager.cc:5484] T ceb177d6952e4c978c8be93b5b026952 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 8 to 9, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:31.687753 13036 raft_consensus.cc:2932] T ceb177d6952e4c978c8be93b5b026952 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } }
I20240503 02:13:31.706221 10470 tablet_service.cc:1514] Processing DeleteTablet for tablet ceb177d6952e4c978c8be93b5b026952 with delete_type TABLET_DATA_TOMBSTONED (TS c6ec2f1283994a3089741715f811c850 not found in new config with opid_index 9) from {username='slave'} at 127.0.0.1:42260
@ 0x7fb4525a4981 kudu::tserver::TSTabletManager::StartTabletCopy()::$_12::operator()() at ??:0
@ 0x7fb4525a470e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
I20240503 02:13:31.717902 17885 tablet_replica.cc:330] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: stopping tablet replica
I20240503 02:13:31.718248 17885 raft_consensus.cc:2229] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:31.719208 17885 raft_consensus.cc:2258] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7fb442ffcb75 kudu::ThreadPool::DispatchThread() at ??:0
I20240503 02:13:31.728745 17885 ts_tablet_manager.cc:1875] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
@ 0x7fb44300019c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fb443000019 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa60bcd std::function<>::operator()() at ??:?
@ 0x7fb442fdb01e kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb4463606db start_thread at ??:0
@ 0x7fb43f19e71f clone at ??:0
I20240503 02:13:31.747823 17885 ts_tablet_manager.cc:1888] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.8
I20240503 02:13:31.747951 17885 log.cc:1196] T ceb177d6952e4c978c8be93b5b026952 P c6ec2f1283994a3089741715f811c850: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-3/wal/wals/ceb177d6952e4c978c8be93b5b026952
I20240503 02:13:31.748898 9936 catalog_manager.cc:4830] TS c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): tablet ceb177d6952e4c978c8be93b5b026952 (table rebalance_test_table_17 [id=46e02feee89d42ec8bc46674e54b22fc]) successfully deleted
I20240503 02:13:31.808349 17444 raft_consensus.cc:1050] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:31.809419 17444 consensus_queue.cc:237] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:31.813077 13040 raft_consensus.cc:1261] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.813809 17444 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.815042 12908 raft_consensus.cc:1261] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.816262 17650 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.818781 10091 raft_consensus.cc:1261] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.819502 17650 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.856685 17650 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:31.869477 9947 catalog_manager.cc:5484] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:31.882056 13302 consensus_queue.cc:237] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:31.882819 10091 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:31.897317 13036 raft_consensus.cc:1261] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:31.900028 17444 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.901540 13036 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:31.918608 17771 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
W20240503 02:13:31.923589 17899 log.cc:924] Time spent T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Append to log took a long time: real 0.069s user 0.000s sys 0.000s
W20240503 02:13:31.925213 10426 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.65:38119, user_credentials={real_user=slave}} blocked reactor thread for 49033.4us
I20240503 02:13:31.932157 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet de58a8fe28a74c28a85c78ea8372ee0a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:31.937947 17898 raft_consensus.cc:1050] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850: attempting to promote NON_VOTER f93b103af083471b8e8c0376e9148aad to VOTER
I20240503 02:13:31.951543 10089 raft_consensus.cc:1261] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:31.955116 17898 consensus_queue.cc:237] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:31.962795 17650 consensus_queue.cc:1035] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:13:31.973737 10424 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 107930us
I20240503 02:13:31.974434 10089 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:31.986584 10226 raft_consensus.cc:1261] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.987282 17905 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:31.997539 13174 raft_consensus.cc:1261] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad [term 1 LEARNER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.998368 13296 raft_consensus.cc:1261] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:31.999275 17898 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:32.031648 17898 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:32.047957 17905 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:32.048898 13296 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:32.050350 13166 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:31.924844 13040 raft_consensus.cc:2932] T de58a8fe28a74c28a85c78ea8372ee0a P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:32.054490 10227 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:31.935214 9947 catalog_manager.cc:5484] T de58a8fe28a74c28a85c78ea8372ee0a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) evicted. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:32.058070 9943 catalog_manager.cc:5484] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, f93b103af083471b8e8c0376e9148aad (127.4.5.71) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:32.086547 17771 raft_consensus.cc:1050] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2: attempting to promote NON_VOTER 031a0afb01b6469299fa829fa3467ef7 to VOTER
I20240503 02:13:32.087517 17771 consensus_queue.cc:237] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:32.087798 17358 raft_consensus.cc:1050] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:32.088732 17358 consensus_queue.cc:237] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:32.093546 10487 consensus_queue.cc:237] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:32.095276 12887 tablet_service.cc:1514] Processing DeleteTablet for tablet de58a8fe28a74c28a85c78ea8372ee0a with delete_type TABLET_DATA_TOMBSTONED (TS e3cbe34992dc4b77b9aa9e9a4c143e78 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:56178
I20240503 02:13:32.096071 13296 raft_consensus.cc:1261] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:32.098495 17932 tablet_replica.cc:330] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: stopping tablet replica
I20240503 02:13:32.098769 17932 raft_consensus.cc:2229] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:32.099104 17932 pending_rounds.cc:62] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Trying to abort 1 pending ops.
I20240503 02:13:32.101933 17898 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:32.117377 10089 raft_consensus.cc:1261] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEARNER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:32.121790 13040 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.111795 13296 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:32.109939 17898 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:32.099170 17932 pending_rounds.cc:68] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Aborting op as it isn't in flight: id { term: 1 index: 3 } timestamp: 7023421078768398336 op_type: CHANGE_CONFIG_OP change_config_record { tablet_id: "de58a8fe28a74c28a85c78ea8372ee0a" old_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } } new_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { replace: true } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } } }
I20240503 02:13:32.127103 17932 raft_consensus.cc:2894] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Aborting config change with OpId 1.3: Aborted: Op aborted
I20240503 02:13:32.127310 17932 raft_consensus.cc:2866] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: CHANGE_CONFIG_OP replication failed: Aborted: Op aborted
I20240503 02:13:32.127437 17932 raft_consensus.cc:2258] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:32.121449 12908 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:13:32.118230 17444 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:32.132673 13174 raft_consensus.cc:1261] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:32.136034 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 274d302b083b42019a6f03305fe516e7 with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:32.139322 17932 ts_tablet_manager.cc:1875] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:32.139897 10227 raft_consensus.cc:1261] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:32.154547 9952 catalog_manager.cc:5484] T 274d302b083b42019a6f03305fe516e7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:32.143724 17650 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:32.174539 17444 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:32.175505 10227 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:32.184674 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet 274d302b083b42019a6f03305fe516e7 with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:32.185050 9952 catalog_manager.cc:5484] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:32.185462 17952 tablet_replica.cc:330] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:32.185706 17952 raft_consensus.cc:2229] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:32.186010 17952 raft_consensus.cc:2258] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:32.122469 17448 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:32.148394 17932 ts_tablet_manager.cc:1888] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:32.187008 17952 ts_tablet_manager.cc:1875] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
I20240503 02:13:32.141419 17904 consensus_queue.cc:1035] T 274d302b083b42019a6f03305fe516e7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:32.181320 10089 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:32.143806 10487 raft_consensus.cc:1261] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:32.416478 17932 log.cc:1196] T de58a8fe28a74c28a85c78ea8372ee0a P e3cbe34992dc4b77b9aa9e9a4c143e78: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-4/wal/wals/de58a8fe28a74c28a85c78ea8372ee0a
I20240503 02:13:32.417793 9936 catalog_manager.cc:4830] TS e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471): tablet de58a8fe28a74c28a85c78ea8372ee0a (table rebalancer_extra_table_22 [id=3ba0db9f05f0466a9fb071f1b4150af6]) successfully deleted
I20240503 02:13:32.424566 17934 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.016s
I20240503 02:13:32.428262 17444 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:32.431221 13174 raft_consensus.cc:2932] T 274d302b083b42019a6f03305fe516e7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } }
I20240503 02:13:32.448076 17942 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
W20240503 02:13:32.451160 12975 connection.cc:535] client connection to 127.4.5.67:41899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.451576 12842 connection.cc:535] client connection to 127.4.5.67:41899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.451565 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.451972 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:32.452631 12908 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
W20240503 02:13:32.458094 10426 connection.cc:535] client connection to 127.4.5.67:41899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.458097 10161 connection.cc:535] client connection to 127.4.5.67:41899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.458451 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.458423 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.460708 13109 connection.cc:535] client connection to 127.4.5.67:41899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.461023 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20240503 02:13:32.466917 9952 catalog_manager.cc:5484] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
W20240503 02:13:32.475594 13245 connection.cc:535] client connection to 127.4.5.67:41899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.475987 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.477208 10028 connection.cc:535] client connection to 127.4.5.67:41899 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.477479 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20240503 02:13:32.488713 17998 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:32.467703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.5.67:41899 (local address 127.4.5.70:56851)
0503 02:13:32.474590 (+ 6887us) negotiation.cc:107] Waiting for socket to connect
0503 02:13:32.474630 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0503 02:13:32.474815 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:13:32.487195 (+ 12380us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":6709,"thread_start_us":7214,"threads_started":1}
W20240503 02:13:32.489059 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.489219 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.489342 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.489825 17995 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:32.460993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.5.67:41899 (local address 127.4.5.66:51963)
0503 02:13:32.475628 (+ 14635us) negotiation.cc:107] Waiting for socket to connect
0503 02:13:32.475654 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0503 02:13:32.475790 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:13:32.488690 (+ 12900us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":14510,"thread_start_us":14387,"threads_started":1}
W20240503 02:13:32.491009 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.491281 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.491468 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.491644 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.492261 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.493065 18001 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:32.478047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.5.67:41899 (local address 127.4.5.72:58883)
0503 02:13:32.481835 (+ 3788us) negotiation.cc:107] Waiting for socket to connect
0503 02:13:32.481860 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0503 02:13:32.482021 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0503 02:13:32.492023 (+ 10002us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":3597,"thread_start_us":5339,"threads_started":1}
W20240503 02:13:32.494204 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.498430 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.498679 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.503126 13040 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:32.505487 10089 consensus_queue.cc:237] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
W20240503 02:13:32.508549 13109 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 47571us
I20240503 02:13:32.508864 12908 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.509658 17934 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:13:32.517060 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.517853 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:32.517452 17934 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:32.521307 12903 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
W20240503 02:13:32.522567 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.523366 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.523803 10028 consensus_peers.cc:485] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.524330 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.525595 12842 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 73677.4us
W20240503 02:13:32.525938 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.527249 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:32.529027 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 5547b33324db46c886287b4ee5035e43 with cas_config_opid_index 8: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:32.535877 9952 catalog_manager.cc:5484] T 5547b33324db46c886287b4ee5035e43 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:32.573297 17952 ts_tablet_manager.cc:1888] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:32.573640 17952 log.cc:1196] T 274d302b083b42019a6f03305fe516e7 P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/274d302b083b42019a6f03305fe516e7
W20240503 02:13:32.581813 17997 negotiation.cc:337] Failed RPC negotiation. Trace:
0503 02:13:32.466172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.4.5.67:41899 (local address 127.4.5.68:41071)
0503 02:13:32.580401 (+114229us) negotiation.cc:107] Waiting for socket to connect
0503 02:13:32.580609 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":114089,"threads_started":1}
I20240503 02:13:32.584163 13036 raft_consensus.cc:1261] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20240503 02:13:32.585320 17942 consensus_queue.cc:1035] T 5547b33324db46c886287b4ee5035e43 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:32.587522 13036 raft_consensus.cc:2932] T 5547b33324db46c886287b4ee5035e43 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 8 to 9, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
W20240503 02:13:32.596727 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.596978 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.597126 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.597280 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.597400 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.597728 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection reset by peer (error 104). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.598706 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:32.600497 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.603456 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet 274d302b083b42019a6f03305fe516e7 (table rebalancer_extra_table_42 [id=1aea76f5c6324651851fa6e313cb7fa0]) successfully deleted
W20240503 02:13:32.604626 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.616983 10491 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:32.641788 10089 consensus_queue.cc:237] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:32.644881 13174 raft_consensus.cc:1261] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20240503 02:13:32.646044 10026 consensus_peers.cc:485] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 -> Peer c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025): Couldn't send request to peer c6ec2f1283994a3089741715f811c850. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42c4f4a0f3d8448aa82becd57e5c49ed. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:32.647301 17934 consensus_queue.cc:1035] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:32.649804 10028 consensus_peers.cc:485] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:32.659972 17942 raft_consensus.cc:2932] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:32.661024 13174 raft_consensus.cc:2932] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } } }
I20240503 02:13:32.669309 9944 catalog_manager.cc:5484] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index -1 to 2, NON_VOTER c6ec2f1283994a3089741715f811c850 (127.4.5.68) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:32.674758 17650 raft_consensus.cc:482] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.675042 17650 raft_consensus.cc:504] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:32.675876 17650 leader_election.cc:288] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:32.677695 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aea508f59f4348b05d93a5afdde4ba" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:32.686746 17942 raft_consensus.cc:482] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.686980 17942 raft_consensus.cc:504] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:32.687711 17942 leader_election.cc:288] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
W20240503 02:13:32.688834 13245 leader_election.cc:334] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.688953 13036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d885837d5e403890617f6ead1fd6c5" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
I20240503 02:13:32.689065 13245 leader_election.cc:302] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082
W20240503 02:13:32.689560 10028 leader_election.cc:334] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.689802 17650 raft_consensus.cc:2726] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:32.689841 10029 leader_election.cc:302] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7; no voters: 47f901e617024baaa1865d12261ef84f, 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:32.690189 17942 raft_consensus.cc:2726] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20240503 02:13:32.769891 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.786154 17934 raft_consensus.cc:482] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.786366 17934 raft_consensus.cc:504] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:32.787132 17934 leader_election.cc:288] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:32.788872 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206e6a3850ff4242b30fa10c3f751fd2" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
W20240503 02:13:32.789690 10028 leader_election.cc:334] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.789809 10028 leader_election.cc:302] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7; no voters: 47f901e617024baaa1865d12261ef84f, 8ee3fd5ad3624ea1b9f681d24e9aabf2
I20240503 02:13:32.790092 17934 raft_consensus.cc:2726] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:32.795130 17728 raft_consensus.cc:482] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.795377 17728 raft_consensus.cc:504] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:32.798099 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20366a3e8dcc4996bfdd1d60ef1be077" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
I20240503 02:13:32.796375 17728 leader_election.cc:288] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
W20240503 02:13:32.800282 12842 leader_election.cc:334] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.800418 12842 leader_election.cc:302] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082
I20240503 02:13:32.800750 17728 raft_consensus.cc:2726] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:32.846894 17803 raft_consensus.cc:482] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.847177 17803 raft_consensus.cc:504] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:32.848238 17803 leader_election.cc:288] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:32.850227 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee635c5c4d2c4befa0283cb8226f28d7" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
W20240503 02:13:32.854354 13109 leader_election.cc:334] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.854542 13109 leader_election.cc:302] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f
I20240503 02:13:32.856123 17803 raft_consensus.cc:2726] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:32.860589 17872 raft_consensus.cc:482] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.860901 17872 raft_consensus.cc:504] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:32.862007 17872 leader_election.cc:288] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
I20240503 02:13:32.866341 12903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20366a3e8dcc4996bfdd1d60ef1be077" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:32.866694 12903 raft_consensus.cc:2454] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:32.867550 10159 leader_election.cc:302] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters:
W20240503 02:13:32.868018 10161 leader_election.cc:334] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.871075 17872 raft_consensus.cc:2781] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:32.871207 17872 raft_consensus.cc:482] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.871315 17872 raft_consensus.cc:3037] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:32.877671 17872 raft_consensus.cc:504] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:32.882966 12903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "20366a3e8dcc4996bfdd1d60ef1be077" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:32.883284 12903 raft_consensus.cc:3037] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:32.888895 17872 leader_election.cc:288] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471)
W20240503 02:13:32.888972 10161 leader_election.cc:334] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.889819 12903 raft_consensus.cc:2454] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 2.
I20240503 02:13:32.890868 10159 leader_election.cc:302] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78, eff8e8d11a4447218741bb994aa41082; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:32.891538 17872 raft_consensus.cc:2781] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:32.891788 17872 raft_consensus.cc:686] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [term 2 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:32.892148 17872 consensus_queue.cc:237] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:32.907029 9944 catalog_manager.cc:5484] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 2 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:32.913151 18066 raft_consensus.cc:482] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.914321 18066 raft_consensus.cc:504] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:32.915253 18066 leader_election.cc:288] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:13:32.916060 10227 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da288f76df0a409eb01ee431d3b83b49" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "eff8e8d11a4447218741bb994aa41082" is_pre_election: true
W20240503 02:13:32.916846 13109 leader_election.cc:334] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.917610 13109 leader_election.cc:302] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f93b103af083471b8e8c0376e9148aad; no voters: 47f901e617024baaa1865d12261ef84f, eff8e8d11a4447218741bb994aa41082
I20240503 02:13:32.918408 18066 raft_consensus.cc:2726] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:32.927788 18011 raft_consensus.cc:482] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.928076 18011 raft_consensus.cc:504] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:32.929265 18011 leader_election.cc:288] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:32.930694 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d885837d5e403890617f6ead1fd6c5" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:32.930999 10089 raft_consensus.cc:2454] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:32.931826 12975 leader_election.cc:302] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b; no voters:
I20240503 02:13:32.932564 18008 raft_consensus.cc:2781] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:32.932672 18008 raft_consensus.cc:482] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.932822 18008 raft_consensus.cc:3037] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:32.936184 18008 raft_consensus.cc:504] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
W20240503 02:13:32.939198 12975 leader_election.cc:334] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.943974 17938 raft_consensus.cc:482] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.944237 17938 raft_consensus.cc:504] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:32.944456 17934 raft_consensus.cc:482] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.944618 17934 raft_consensus.cc:504] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:32.945113 17938 leader_election.cc:288] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:32.945286 17934 leader_election.cc:288] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
W20240503 02:13:32.948004 10028 leader_election.cc:334] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:32.949613 13245 leader_election.cc:334] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.951889 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "90d885837d5e403890617f6ead1fd6c5" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:32.952117 10091 raft_consensus.cc:3037] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:32.955196 10091 raft_consensus.cc:2454] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 2.
I20240503 02:13:32.956158 17898 raft_consensus.cc:482] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.956288 12975 leader_election.cc:302] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8d454a3607d042dc9559ed59c3ccd84b; no voters:
I20240503 02:13:32.956486 17898 raft_consensus.cc:504] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:32.956925 18006 raft_consensus.cc:2781] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:32.957063 18006 raft_consensus.cc:686] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:32.957433 17898 leader_election.cc:288] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:32.957393 18006 consensus_queue.cc:237] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:13:32.957792 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:32.959105 13245 leader_election.cc:302] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f
I20240503 02:13:32.959911 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:32.960240 10491 raft_consensus.cc:2454] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
I20240503 02:13:32.961429 17650 raft_consensus.cc:2726] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:32.961560 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:32.961792 10089 raft_consensus.cc:2454] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:32.963750 18008 leader_election.cc:288] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:32.966761 17938 raft_consensus.cc:482] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.967029 17938 raft_consensus.cc:504] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:32.967834 17938 leader_election.cc:288] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:13:32.969617 10426 leader_election.cc:302] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters:
I20240503 02:13:32.971582 13036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4b31cd7334ed6b18288c1b6d3f958" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8d454a3607d042dc9559ed59c3ccd84b" is_pre_election: true
W20240503 02:13:32.971879 13245 leader_election.cc:334] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.972807 17898 raft_consensus.cc:2781] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:32.972946 17898 raft_consensus.cc:482] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.973068 17898 raft_consensus.cc:3037] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:32.973937 13246 leader_election.cc:302] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f, 8d454a3607d042dc9559ed59c3ccd84b
I20240503 02:13:32.974957 17444 raft_consensus.cc:2726] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:32.979283 17444 raft_consensus.cc:482] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.979521 17444 raft_consensus.cc:504] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:32.979688 17898 raft_consensus.cc:504] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:32.980278 17444 leader_election.cc:288] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
W20240503 02:13:32.981832 10426 leader_election.cc:334] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.982708 10026 leader_election.cc:302] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:32.983336 17942 raft_consensus.cc:2781] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:32.983465 17942 raft_consensus.cc:482] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.983603 17942 raft_consensus.cc:3037] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:32.988193 17942 raft_consensus.cc:504] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
W20240503 02:13:32.990646 12975 leader_election.cc:334] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.991775 17938 raft_consensus.cc:482] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:32.991977 17938 raft_consensus.cc:504] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:32.992533 17942 leader_election.cc:288] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: Requested vote from peers c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:32.993122 17938 leader_election.cc:288] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:32.993441 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390ad47a9a534abb81d59dc7604c72d3" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:32.995509 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206e6a3850ff4242b30fa10c3f751fd2" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:32.995739 10089 raft_consensus.cc:2454] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 1.
W20240503 02:13:32.995857 10426 leader_election.cc:334] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:32.996589 10487 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850"
I20240503 02:13:32.996963 10487 raft_consensus.cc:2379] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 031a0afb01b6469299fa829fa3467ef7 in current term 2: Already voted for candidate c6ec2f1283994a3089741715f811c850 in this term.
I20240503 02:13:32.998850 13296 consensus_queue.cc:237] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:33.000954 9952 catalog_manager.cc:5484] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 2 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } }
I20240503 02:13:33.002707 17898 leader_election.cc:288] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:33.003345 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:33.003696 10089 raft_consensus.cc:2379] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate c6ec2f1283994a3089741715f811c850 in current term 2: Already voted for candidate 031a0afb01b6469299fa829fa3467ef7 in this term.
I20240503 02:13:33.004588 13245 leader_election.cc:302] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters:
I20240503 02:13:33.002799 10487 raft_consensus.cc:1261] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:33.005779 17444 raft_consensus.cc:2781] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.005888 17444 raft_consensus.cc:482] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.005987 17444 raft_consensus.cc:3037] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.005146 10426 leader_election.cc:302] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6ec2f1283994a3089741715f811c850; no voters: 031a0afb01b6469299fa829fa3467ef7, 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.007884 17903 raft_consensus.cc:2726] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:13:33.009297 17938 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:33.016347 17650 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:33.019140 10091 raft_consensus.cc:1261] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
W20240503 02:13:33.020875 10028 leader_election.cc:334] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.021028 10028 leader_election.cc:302] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7; no voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850
I20240503 02:13:33.021308 17942 raft_consensus.cc:2726] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20240503 02:13:33.021090 10491 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:33.023895 17444 raft_consensus.cc:504] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.028951 17444 leader_election.cc:288] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 election: Requested vote from peers 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:33.032990 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet f38186393ba043a19f834d1d8780053a with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:33.034464 9944 catalog_manager.cc:5484] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:33.036129 13245 leader_election.cc:334] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:33.036573 13245 leader_election.cc:334] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.036728 13245 leader_election.cc:302] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850
I20240503 02:13:33.037003 17444 raft_consensus.cc:2726] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:33.038061 10089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "206e6a3850ff4242b30fa10c3f751fd2" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:33.038370 10089 raft_consensus.cc:3037] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.038578 17444 consensus_queue.cc:1035] T f38186393ba043a19f834d1d8780053a P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:33.040913 17903 raft_consensus.cc:482] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.041101 17903 raft_consensus.cc:504] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
W20240503 02:13:33.041404 13245 leader_election.cc:334] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.041931 17903 leader_election.cc:288] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.042793 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390ad47a9a534abb81d59dc7604c72d3" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:33.043076 13296 raft_consensus.cc:2454] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
W20240503 02:13:33.043526 10426 leader_election.cc:334] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.043639 10426 leader_election.cc:302] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.043889 17903 raft_consensus.cc:2781] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.043959 17903 raft_consensus.cc:482] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.044023 17903 raft_consensus.cc:3037] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.045755 10089 raft_consensus.cc:2454] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8ee3fd5ad3624ea1b9f681d24e9aabf2 in term 2.
I20240503 02:13:33.046424 13245 leader_election.cc:302] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.046610 17903 raft_consensus.cc:504] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.047519 10207 tablet_service.cc:1514] Processing DeleteTablet for tablet f38186393ba043a19f834d1d8780053a with delete_type TABLET_DATA_TOMBSTONED (TS eff8e8d11a4447218741bb994aa41082 not found in new config with opid_index 4) from {username='slave'} at 127.0.0.1:41072
I20240503 02:13:33.047950 17952 tablet_replica.cc:330] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: stopping tablet replica
I20240503 02:13:33.048214 17952 raft_consensus.cc:2229] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus shutting down.
I20240503 02:13:33.048590 17952 raft_consensus.cc:2258] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Raft consensus is shut down!
I20240503 02:13:33.050163 17952 ts_tablet_manager.cc:1875] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: Deleting tablet data with delete state TABLET_DATA_TOMBSTONED
W20240503 02:13:33.054231 10426 leader_election.cc:334] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.054529 10091 raft_consensus.cc:2932] T f38186393ba043a19f834d1d8780053a P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } } }
I20240503 02:13:33.056769 17650 raft_consensus.cc:2781] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.058328 17650 raft_consensus.cc:686] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 LEADER]: Becoming Leader. State: Replica: 8ee3fd5ad3624ea1b9f681d24e9aabf2, State: Running, Role: LEADER
I20240503 02:13:33.058786 17650 consensus_queue.cc:237] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.073432 17952 ts_tablet_manager.cc:1888] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: tablet deleted with delete type TABLET_DATA_TOMBSTONED: last-logged OpId 1.3
I20240503 02:13:33.073628 17952 log.cc:1196] T f38186393ba043a19f834d1d8780053a P eff8e8d11a4447218741bb994aa41082: Deleting WAL directory at /tmp/dist-test-task48YlKh/test-tmp/rebalancer_tool-test.1.DDLDuringRebalancingTest.TablesCreatedAndDeletedDuringRebalancing_1.1714702282720633-4117-0/raft_consensus-itest-cluster/ts-1/wal/wals/f38186393ba043a19f834d1d8780053a
I20240503 02:13:33.074760 9938 catalog_manager.cc:4830] TS eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): tablet f38186393ba043a19f834d1d8780053a (table rebalancer_extra_table_36 [id=0f99899d7eb64df5b1c085b833865e94]) successfully deleted
I20240503 02:13:33.078635 17903 leader_election.cc:288] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.080562 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "390ad47a9a534abb81d59dc7604c72d3" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:33.080870 13296 raft_consensus.cc:3037] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.081025 17650 raft_consensus.cc:482] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.081193 17650 raft_consensus.cc:504] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:33.081931 17650 leader_election.cc:288] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:33.082435 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33490610e3b4819a61ef4bbfe0c5942" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
W20240503 02:13:33.083698 13245 leader_election.cc:334] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.083828 13245 leader_election.cc:302] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f, f93b103af083471b8e8c0376e9148aad
I20240503 02:13:33.083971 13296 raft_consensus.cc:2454] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 2.
I20240503 02:13:33.084092 17444 raft_consensus.cc:2726] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:33.084791 10424 leader_election.cc:302] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.085085 17898 raft_consensus.cc:2781] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.085320 17898 raft_consensus.cc:686] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [term 2 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:33.085599 17898 consensus_queue.cc:237] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.091692 9952 catalog_manager.cc:5484] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 2 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.094400 18006 raft_consensus.cc:482] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.094640 18006 raft_consensus.cc:504] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.095563 18006 leader_election.cc:288] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
W20240503 02:13:33.098093 12975 leader_election.cc:334] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.098670 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4b31cd7334ed6b18288c1b6d3f958" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:33.098783 9944 catalog_manager.cc:5484] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72). New cstate: current_term: 2 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:33.099440 13296 raft_consensus.cc:2454] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 1.
I20240503 02:13:33.100239 12973 leader_election.cc:302] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.100726 18006 raft_consensus.cc:2781] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.100859 18006 raft_consensus.cc:482] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.100996 18006 raft_consensus.cc:3037] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.103989 18006 raft_consensus.cc:504] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
W20240503 02:13:33.106921 12975 leader_election.cc:334] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.107913 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d2d4b31cd7334ed6b18288c1b6d3f958" candidate_uuid: "8d454a3607d042dc9559ed59c3ccd84b" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:33.108178 13302 raft_consensus.cc:3037] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.111613 18082 ts_tablet_manager.cc:902] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:33.111642 13302 raft_consensus.cc:2454] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8d454a3607d042dc9559ed59c3ccd84b in term 2.
I20240503 02:13:33.113010 18006 leader_election.cc:288] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.116466 12973 leader_election.cc:302] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8d454a3607d042dc9559ed59c3ccd84b, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.116865 18008 raft_consensus.cc:2781] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.117081 18008 raft_consensus.cc:686] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [term 2 LEADER]: Becoming Leader. State: Replica: 8d454a3607d042dc9559ed59c3ccd84b, State: Running, Role: LEADER
I20240503 02:13:33.117380 18008 consensus_queue.cc:237] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.117830 17934 raft_consensus.cc:482] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.118072 17934 raft_consensus.cc:504] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.119041 17934 leader_election.cc:288] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.120074 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:33.120508 13302 raft_consensus.cc:2454] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
W20240503 02:13:33.120817 10028 leader_election.cc:334] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.121006 10026 leader_election.cc:302] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.122113 17934 raft_consensus.cc:2781] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.122210 17934 raft_consensus.cc:482] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.122279 17934 raft_consensus.cc:3037] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.123854 18082 tablet_copy_client.cc:308] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:13:33.124032 9944 catalog_manager.cc:5484] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70). New cstate: current_term: 2 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.124706 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet 42c4f4a0f3d8448aa82becd57e5c49ed from peer c6ec2f1283994a3089741715f811c850 ({username='slave'} at 127.4.5.68:36597)
I20240503 02:13:33.124867 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet 42c4f4a0f3d8448aa82becd57e5c49ed from peer c6ec2f1283994a3089741715f811c850 at {username='slave'} at 127.4.5.68:36597: session id = c6ec2f1283994a3089741715f811c850-42c4f4a0f3d8448aa82becd57e5c49ed
I20240503 02:13:33.125732 17934 raft_consensus.cc:504] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.127852 10115 tablet_copy_source_session.cc:205] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:13:33.130235 17934 leader_election.cc:288] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.131223 17393 raft_consensus.cc:482] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.131356 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "12da2248f416404da29a4557b90daef7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:33.131588 13302 raft_consensus.cc:3037] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.131443 17393 raft_consensus.cc:504] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:33.132308 17393 leader_election.cc:288] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
W20240503 02:13:33.132599 10028 leader_election.cc:334] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.134919 18082 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42c4f4a0f3d8448aa82becd57e5c49ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20240503 02:13:33.134835 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da288f76df0a409eb01ee431d3b83b49" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:33.135021 13302 raft_consensus.cc:2454] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 2.
I20240503 02:13:33.135154 13174 raft_consensus.cc:2454] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
W20240503 02:13:33.136309 10161 leader_election.cc:334] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.136255 17373 raft_consensus.cc:482] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.136790 10162 leader_election.cc:302] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eff8e8d11a4447218741bb994aa41082, f93b103af083471b8e8c0376e9148aad; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.137157 10026 leader_election.cc:302] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.137463 17934 raft_consensus.cc:2781] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.138598 17373 raft_consensus.cc:504] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:33.137652 17934 raft_consensus.cc:686] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [term 2 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:13:33.139845 17373 leader_election.cc:288] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:33.139835 17934 consensus_queue.cc:237] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.140735 17872 raft_consensus.cc:2781] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.140859 17872 raft_consensus.cc:482] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.140974 17872 raft_consensus.cc:3037] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.141244 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:33.144133 17872 raft_consensus.cc:504] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
W20240503 02:13:33.147549 10161 leader_election.cc:334] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.148334 17872 leader_election.cc:288] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:33.149056 18082 tablet_copy_client.cc:783] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 0 data blocks...
I20240503 02:13:33.149402 18082 tablet_copy_client.cc:647] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: tablet copy: Starting download of 1 WAL segments...
I20240503 02:13:33.150410 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "da288f76df0a409eb01ee431d3b83b49" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:33.150774 13174 raft_consensus.cc:3037] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.151635 18082 tablet_copy_client.cc:523] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: tablet copy: Tablet Copy complete. Replacing tablet superblock.
W20240503 02:13:33.154206 12842 leader_election.cc:334] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.154428 12842 leader_election.cc:302] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850
I20240503 02:13:33.155086 17373 raft_consensus.cc:2726] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20240503 02:13:33.158704 18082 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: Bootstrap starting.
I20240503 02:13:33.168030 13174 raft_consensus.cc:2454] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 2.
I20240503 02:13:33.169575 10162 leader_election.cc:302] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: eff8e8d11a4447218741bb994aa41082, f93b103af083471b8e8c0376e9148aad; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.170329 17872 raft_consensus.cc:2781] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.170567 17872 raft_consensus.cc:686] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [term 2 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:33.170989 17872 consensus_queue.cc:237] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:33.175572 18082 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=0 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20240503 02:13:33.176062 18082 tablet_bootstrap.cc:492] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: Bootstrap complete.
I20240503 02:13:33.176420 18082 ts_tablet_manager.cc:1367] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.005s
I20240503 02:13:33.177170 18082 raft_consensus.cc:348] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:33.177407 18082 raft_consensus.cc:729] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: c6ec2f1283994a3089741715f811c850, State: Initialized, Role: LEARNER
I20240503 02:13:33.177800 18082 consensus_queue.cc:260] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: NON_VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: true } }
I20240503 02:13:33.180590 18082 ts_tablet_manager.cc:1398] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20240503 02:13:33.183245 18059 raft_consensus.cc:482] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.183470 18059 raft_consensus.cc:504] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:33.184485 10115 tablet_copy_service.cc:342] P 031a0afb01b6469299fa829fa3467ef7: Request end of tablet copy session c6ec2f1283994a3089741715f811c850-42c4f4a0f3d8448aa82becd57e5c49ed received from {username='slave'} at 127.4.5.68:36597
I20240503 02:13:33.184624 10115 tablet_copy_service.cc:433] P 031a0afb01b6469299fa829fa3467ef7: ending tablet copy session c6ec2f1283994a3089741715f811c850-42c4f4a0f3d8448aa82becd57e5c49ed on tablet 42c4f4a0f3d8448aa82becd57e5c49ed with peer c6ec2f1283994a3089741715f811c850
I20240503 02:13:33.186333 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33490610e3b4819a61ef4bbfe0c5942" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:33.186594 18059 leader_election.cc:288] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:33.186653 13302 raft_consensus.cc:2454] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 1.
I20240503 02:13:33.187192 13107 leader_election.cc:302] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters:
I20240503 02:13:33.187508 18059 raft_consensus.cc:2781] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.187709 18059 raft_consensus.cc:482] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.187902 18059 raft_consensus.cc:3037] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:33.190882 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.191716 18059 raft_consensus.cc:504] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
W20240503 02:13:33.192356 13109 leader_election.cc:334] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.192348 9952 catalog_manager.cc:5484] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 2 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.194767 17898 raft_consensus.cc:482] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.194842 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b33490610e3b4819a61ef4bbfe0c5942" candidate_uuid: "f93b103af083471b8e8c0376e9148aad" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:33.194936 17898 raft_consensus.cc:504] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:33.195077 13302 raft_consensus.cc:3037] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:33.195108 13109 leader_election.cc:334] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.195326 18059 leader_election.cc:288] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 election: Requested vote from peers 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:33.195734 17898 leader_election.cc:288] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:33.195794 17872 raft_consensus.cc:482] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.195997 17872 raft_consensus.cc:504] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:33.196861 17872 leader_election.cc:288] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.197453 12903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" is_pre_election: true
I20240503 02:13:33.197731 12903 raft_consensus.cc:2454] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 2.
I20240503 02:13:33.198112 13302 raft_consensus.cc:2454] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f93b103af083471b8e8c0376e9148aad in term 2.
I20240503 02:13:33.198638 13296 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aea508f59f4348b05d93a5afdde4ba" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
W20240503 02:13:33.198762 10426 leader_election.cc:334] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.198879 13296 raft_consensus.cc:2454] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 1.
I20240503 02:13:33.199254 13107 leader_election.cc:302] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, f93b103af083471b8e8c0376e9148aad; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.200011 10424 leader_election.cc:302] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.200176 18059 raft_consensus.cc:2781] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.200477 17898 raft_consensus.cc:2781] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:13:33.200486 18059 raft_consensus.cc:686] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [term 2 LEADER]: Becoming Leader. State: Replica: f93b103af083471b8e8c0376e9148aad, State: Running, Role: LEADER
I20240503 02:13:33.200564 17898 raft_consensus.cc:482] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.200651 17898 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:33.200927 10159 leader_election.cc:302] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082; no voters:
I20240503 02:13:33.200866 18059 consensus_queue.cc:237] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } }
I20240503 02:13:33.202087 17872 raft_consensus.cc:2781] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.202193 17872 raft_consensus.cc:482] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.202280 17872 raft_consensus.cc:3037] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.204154 17898 raft_consensus.cc:504] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:33.204775 9944 catalog_manager.cc:5484] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 2 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.205334 17872 raft_consensus.cc:504] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:33.205991 12903 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "db2da39193164d1ebf830282258d47d7" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 3 candidate_status { last_received { term: 2 index: 8 } } ignore_live_leader: false dest_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78"
I20240503 02:13:33.206382 12903 raft_consensus.cc:3037] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:13:33.210625 10161 leader_election.cc:334] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.211566 12903 raft_consensus.cc:2454] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 3.
I20240503 02:13:33.212185 10424 leader_election.cc:302] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6ec2f1283994a3089741715f811c850, e3cbe34992dc4b77b9aa9e9a4c143e78; no voters:
W20240503 02:13:33.212229 10161 leader_election.cc:334] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.212810 17903 raft_consensus.cc:2781] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:33.212980 17903 raft_consensus.cc:686] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [term 3 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:33.212942 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35aea508f59f4348b05d93a5afdde4ba" candidate_uuid: "eff8e8d11a4447218741bb994aa41082" candidate_term: 2 candidate_status { last_received { term: 1 index: 7 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:33.213282 13302 raft_consensus.cc:3037] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.213323 17903 consensus_queue.cc:237] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 8, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:33.213662 17898 leader_election.cc:288] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 election: Requested vote from peers e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69:33471), 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899)
I20240503 02:13:33.216197 17872 leader_election.cc:288] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.217008 13302 raft_consensus.cc:2454] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eff8e8d11a4447218741bb994aa41082 in term 2.
W20240503 02:13:33.217542 10426 leader_election.cc:334] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.217557 10159 leader_election.cc:302] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, eff8e8d11a4447218741bb994aa41082; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.217931 17872 raft_consensus.cc:2781] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.218118 17872 raft_consensus.cc:686] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 2 LEADER]: Becoming Leader. State: Replica: eff8e8d11a4447218741bb994aa41082, State: Running, Role: LEADER
I20240503 02:13:33.217993 9944 catalog_manager.cc:5484] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to f93b103af083471b8e8c0376e9148aad (127.4.5.71). New cstate: current_term: 2 leader_uuid: "f93b103af083471b8e8c0376e9148aad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.218717 17872 consensus_queue.cc:237] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } }
I20240503 02:13:33.231892 17650 raft_consensus.cc:482] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.232201 17650 raft_consensus.cc:504] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.232280 9944 catalog_manager.cc:5484] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to eff8e8d11a4447218741bb994aa41082 (127.4.5.66). New cstate: current_term: 2 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.234304 17650 leader_election.cc:288] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), c6ec2f1283994a3089741715f811c850 (127.4.5.68:35025)
I20240503 02:13:33.235141 10491 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "c6ec2f1283994a3089741715f811c850" is_pre_election: true
I20240503 02:13:33.238581 17904 raft_consensus.cc:482] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.238857 17904 raft_consensus.cc:504] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.239811 17904 leader_election.cc:288] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.241727 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" is_pre_election: true
I20240503 02:13:33.242007 17358 raft_consensus.cc:482] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.242056 13302 raft_consensus.cc:2454] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 1.
I20240503 02:13:33.242200 17358 raft_consensus.cc:504] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:33.242978 17358 leader_election.cc:288] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
W20240503 02:13:33.243680 10426 leader_election.cc:334] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.244158 10424 leader_election.cc:302] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.244074 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee635c5c4d2c4befa0283cb8226f28d7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad" is_pre_election: true
I20240503 02:13:33.244495 13174 raft_consensus.cc:2454] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 1.
W20240503 02:13:33.244817 10028 leader_election.cc:334] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.245033 10029 leader_election.cc:302] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.245959 17358 raft_consensus.cc:2781] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.246096 17358 raft_consensus.cc:482] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.246192 17358 raft_consensus.cc:3037] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.246559 17904 raft_consensus.cc:2781] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20240503 02:13:33.246681 17904 raft_consensus.cc:482] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47f901e617024baaa1865d12261ef84f)
I20240503 02:13:33.246771 17904 raft_consensus.cc:3037] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.248347 17358 raft_consensus.cc:504] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
W20240503 02:13:33.246218 13245 leader_election.cc:334] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.248914 13245 leader_election.cc:302] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2; no voters: 47f901e617024baaa1865d12261ef84f, c6ec2f1283994a3089741715f811c850
I20240503 02:13:33.249807 17650 raft_consensus.cc:2726] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20240503 02:13:33.250375 17904 raft_consensus.cc:504] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
W20240503 02:13:33.251008 10028 leader_election.cc:334] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.252148 13174 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee635c5c4d2c4befa0283cb8226f28d7" candidate_uuid: "031a0afb01b6469299fa829fa3467ef7" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "f93b103af083471b8e8c0376e9148aad"
I20240503 02:13:33.252527 13174 raft_consensus.cc:3037] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Advancing to term 2
W20240503 02:13:33.253278 10426 leader_election.cc:334] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:33.253871 9944 catalog_manager.cc:5484] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 2 to 3, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 3 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:33.254424 17904 leader_election.cc:288] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:13:33.255116 13302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c762709f66f44b98089e69cbace1570" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 2 candidate_status { last_received { term: 1 index: 6 } } ignore_live_leader: false dest_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2"
I20240503 02:13:33.255353 13302 raft_consensus.cc:3037] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Advancing to term 2
I20240503 02:13:33.258535 13174 raft_consensus.cc:2454] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 031a0afb01b6469299fa829fa3467ef7 in term 2.
I20240503 02:13:33.258927 13302 raft_consensus.cc:2454] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 2.
I20240503 02:13:33.259181 10029 leader_election.cc:302] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, f93b103af083471b8e8c0376e9148aad; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.259553 17934 raft_consensus.cc:2781] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.259706 17934 raft_consensus.cc:686] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [term 2 LEADER]: Becoming Leader. State: Replica: 031a0afb01b6469299fa829fa3467ef7, State: Running, Role: LEADER
I20240503 02:13:33.259778 17358 leader_election.cc:288] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [CANDIDATE]: Term 2 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557)
I20240503 02:13:33.260002 17934 consensus_queue.cc:237] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } }
I20240503 02:13:33.260350 10424 leader_election.cc:302] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8ee3fd5ad3624ea1b9f681d24e9aabf2, c6ec2f1283994a3089741715f811c850; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:33.261375 17904 raft_consensus.cc:2781] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader election won for term 2
I20240503 02:13:33.261636 17904 raft_consensus.cc:686] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [term 2 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:33.262145 17904 consensus_queue.cc:237] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } }
I20240503 02:13:33.268407 9944 catalog_manager.cc:5484] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 2 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.270942 9952 catalog_manager.cc:5484] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: term changed from 1 to 2, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65). New cstate: current_term: 2 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
I20240503 02:13:33.307015 12903 raft_consensus.cc:1261] T 20366a3e8dcc4996bfdd1d60ef1be077 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:33.307793 17872 consensus_queue.cc:1035] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:13:33.313292 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:33.451828 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.469986 10091 raft_consensus.cc:1261] T 90d885837d5e403890617f6ead1fd6c5 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:13:33.470762 18006 consensus_queue.cc:1035] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:13:33.488796 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.510273 10487 raft_consensus.cc:1203] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.1->[1.2-1.2] Dedup: 1.2->[]
I20240503 02:13:33.541606 10091 raft_consensus.cc:1261] T 206e6a3850ff4242b30fa10c3f751fd2 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:33.542371 17650 consensus_queue.cc:1035] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:33.554486 13302 raft_consensus.cc:1261] T 390ad47a9a534abb81d59dc7604c72d3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:33.555282 17904 consensus_queue.cc:1035] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:13:33.578032 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:33.608178 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:33.618582 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:33.620093 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.629174 13302 raft_consensus.cc:1261] T 12da2248f416404da29a4557b90daef7 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:33.629884 17934 consensus_queue.cc:1035] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:33.639964 13302 raft_consensus.cc:1261] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20240503 02:13:33.642053 18008 consensus_queue.cc:1035] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:33.670336 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.683509 13302 raft_consensus.cc:1261] T 5c762709f66f44b98089e69cbace1570 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:33.684206 17903 consensus_queue.cc:1035] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20240503 02:13:33.689960 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:33.745234 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:33.749300 10161 consensus_peers.cc:485] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:33.755357 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.775864 13302 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:33.776633 17872 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:33.794843 12903 raft_consensus.cc:3037] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 LEARNER]: Advancing to term 2
I20240503 02:13:33.798192 12903 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 2 index: 8. (index mismatch)
I20240503 02:13:33.799149 17393 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: NON_VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:13:33.802076 13174 raft_consensus.cc:1261] T da288f76df0a409eb01ee431d3b83b49 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:33.804299 17393 consensus_queue.cc:1035] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:33.807318 13296 raft_consensus.cc:1261] T b33490610e3b4819a61ef4bbfe0c5942 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20240503 02:13:33.808106 18059 consensus_queue.cc:1035] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:13:33.813895 17393 raft_consensus.cc:1050] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082: attempting to promote NON_VOTER e3cbe34992dc4b77b9aa9e9a4c143e78 to VOTER
I20240503 02:13:33.814922 17393 consensus_queue.cc:237] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 2.8, Last appended by leader: 7, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:33.817901 12908 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 LEARNER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:33.818586 18020 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:33.820124 13302 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 2 index: 9. (index mismatch)
I20240503 02:13:33.835156 12908 raft_consensus.cc:1261] T db2da39193164d1ebf830282258d47d7 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 3 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 2 index: 8. Preceding OpId from leader: term: 3 index: 9. (index mismatch)
I20240503 02:13:33.837782 13166 raft_consensus.cc:1261] T ee635c5c4d2c4befa0283cb8226f28d7 P f93b103af083471b8e8c0376e9148aad [term 2 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20240503 02:13:33.839468 17942 consensus_queue.cc:1035] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:13:33.842036 17872 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20240503 02:13:33.843426 17903 consensus_queue.cc:1035] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
W20240503 02:13:33.849241 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:33.857268 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:33.861510 10161 consensus_peers.cc:485] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:33.864564 17872 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 2 LEADER]: Committing config change with OpId 2.9: config changed from index 7 to 9, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:33.868644 13296 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:33.918686 9951 catalog_manager.cc:5484] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 7 to 9, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New cstate: current_term: 2 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:33.918754 12908 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.9: config changed from index 7 to 9, e3cbe34992dc4b77b9aa9e9a4c143e78 (127.4.5.69) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:33.971164 10227 consensus_queue.cc:237] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 9, Committed index: 9, Last appended: 2.9, Last appended by leader: 7, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } }
I20240503 02:13:33.983116 12908 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:33.985606 13296 raft_consensus.cc:1261] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 2 index: 9. Preceding OpId from leader: term: 2 index: 10. (index mismatch)
I20240503 02:13:33.986825 17393 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.987438 17393 consensus_queue.cc:1035] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 10, Last known committed idx: 9, Time since last communication: 0.000s
I20240503 02:13:33.991138 18020 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 [term 2 LEADER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:33.991971 13296 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:33.997637 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 35aea508f59f4348b05d93a5afdde4ba with cas_config_opid_index 9: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:34.001401 12908 raft_consensus.cc:2932] T 35aea508f59f4348b05d93a5afdde4ba P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 2 FOLLOWER]: Committing config change with OpId 2.10: config changed from index 9 to 10, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } }
I20240503 02:13:34.006688 9947 catalog_manager.cc:5484] T 35aea508f59f4348b05d93a5afdde4ba P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 9 to 10, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 2 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 10 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
I20240503 02:13:34.064353 17942 raft_consensus.cc:1050] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7: attempting to promote NON_VOTER c6ec2f1283994a3089741715f811c850 to VOTER
I20240503 02:13:34.065534 17942 consensus_queue.cc:237] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:34.070127 13166 raft_consensus.cc:1261] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:34.070281 10491 raft_consensus.cc:1261] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 [term 1 LEARNER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:13:34.070984 17934 consensus_queue.cc:1035] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
W20240503 02:13:34.071462 10028 consensus_peers.cc:485] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:13:34.071512 18171 consensus_queue.cc:1035] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:13:34.074388 17934 raft_consensus.cc:2932] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:34.075115 10491 raft_consensus.cc:2932] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:34.079706 13166 raft_consensus.cc:2932] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:34.083220 9944 catalog_manager.cc:5484] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 reported cstate change: config changed from index 2 to 3, c6ec2f1283994a3089741715f811c850 (127.4.5.68) changed from NON_VOTER to VOTER. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { replace: true } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:34.091284 10091 consensus_queue.cc:237] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } }
I20240503 02:13:34.093907 13166 raft_consensus.cc:1261] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:34.094234 10491 raft_consensus.cc:1261] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:13:34.094534 17942 consensus_queue.cc:1035] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:34.094782 17358 consensus_queue.cc:1035] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:13:34.097517 18171 raft_consensus.cc:2932] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:34.098253 13166 raft_consensus.cc:2932] T 42c4f4a0f3d8448aa82becd57e5c49ed P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:34.098263 10491 raft_consensus.cc:2932] T 42c4f4a0f3d8448aa82becd57e5c49ed P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } }
I20240503 02:13:34.101554 9938 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet 42c4f4a0f3d8448aa82becd57e5c49ed with cas_config_opid_index 3: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:13:34.106590 9951 catalog_manager.cc:5484] T 42c4f4a0f3d8448aa82becd57e5c49ed P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 3 to 4, VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } }
W20240503 02:13:34.162472 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:34.429675 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.495170 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.508499 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.555660 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:34.560681 18131 raft_consensus.cc:482] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20240503 02:13:34.560950 18131 raft_consensus.cc:504] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:34.561678 18131 leader_election.cc:288] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:34.562420 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7" is_pre_election: true
I20240503 02:13:34.562685 10091 raft_consensus.cc:2454] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 2.
W20240503 02:13:34.563585 10426 leader_election.cc:334] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:34.563861 10426 leader_election.cc:302] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:34.564116 18131 raft_consensus.cc:2781] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20240503 02:13:34.564252 18131 raft_consensus.cc:482] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20240503 02:13:34.564360 18131 raft_consensus.cc:3037] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 2 FOLLOWER]: Advancing to term 3
I20240503 02:13:34.566727 18131 raft_consensus.cc:504] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
I20240503 02:13:34.567371 18131 leader_election.cc:288] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 election: Requested vote from peers 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899), 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:13:34.568058 10091 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c206318e0524101a4daee638813e041" candidate_uuid: "c6ec2f1283994a3089741715f811c850" candidate_term: 3 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "031a0afb01b6469299fa829fa3467ef7"
I20240503 02:13:34.568281 10091 raft_consensus.cc:3037] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 2 FOLLOWER]: Advancing to term 3
W20240503 02:13:34.568565 10426 leader_election.cc:334] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:13:34.571189 10091 raft_consensus.cc:2454] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c6ec2f1283994a3089741715f811c850 in term 3.
I20240503 02:13:34.571583 10426 leader_election.cc:302] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 031a0afb01b6469299fa829fa3467ef7, c6ec2f1283994a3089741715f811c850; no voters: 47f901e617024baaa1865d12261ef84f
I20240503 02:13:34.571873 18131 raft_consensus.cc:2781] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 3 FOLLOWER]: Leader election won for term 3
I20240503 02:13:34.572043 18131 raft_consensus.cc:686] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [term 3 LEADER]: Becoming Leader. State: Replica: c6ec2f1283994a3089741715f811c850, State: Running, Role: LEADER
I20240503 02:13:34.572353 18131 consensus_queue.cc:237] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } }
W20240503 02:13:34.575181 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:34.577616 9951 catalog_manager.cc:5484] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 reported cstate change: term changed from 1 to 3, leader changed from 47f901e617024baaa1865d12261ef84f (127.4.5.67) to c6ec2f1283994a3089741715f811c850 (127.4.5.68). New cstate: current_term: 3 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false } health_report { overall_health: UNKNOWN } } }
W20240503 02:13:34.581630 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.595108 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:34.621083 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.659685 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.683782 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.703971 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.707379 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.790138 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.846643 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.850927 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.862728 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.926949 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:34.941695 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:34.981632 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.022164 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.025540 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.036345 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.089370 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.095517 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:13:35.097222 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.107832 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20240503 02:13:35.172700 10091 raft_consensus.cc:1261] T 4c206318e0524101a4daee638813e041 P 031a0afb01b6469299fa829fa3467ef7 [term 3 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 5. (index mismatch)
I20240503 02:13:35.173470 18131 consensus_queue.cc:1035] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:13:35.210309 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:35.284574 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.706010 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:35.875538 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.900210 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:35.982631 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.013957 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.045958 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.046465 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.106227 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.259624 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.266144 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:36.278990 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.327054 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:36.337553 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.450423 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:36.869463 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:36.895573 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:36.895732 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:36.896787 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.019652 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.023444 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.125874 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.126965 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.215117 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.246595 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.304167 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.326292 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.346869 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.356581 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.357872 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.437914 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.449952 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.482263 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 77 similar messages]
W20240503 02:13:37.485648 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:13:37.486730 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:37.526279 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 55 similar messages]
W20240503 02:13:37.527782 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.542613 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 104 similar messages]
W20240503 02:13:37.547240 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.574263 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:13:37.575238 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20240503 02:13:37.588519 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.590273 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.592005 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:13:37.597844 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.724313 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.725071 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:37.754761 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 22 similar messages]
W20240503 02:13:37.921859 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:37.975230 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.305403 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.345474 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 24 similar messages]
W20240503 02:13:38.346915 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:38.419054 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:38.461191 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.548386 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.653805 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.664611 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.689502 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.706637 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.716334 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.723680 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.732635 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.777890 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:38.824246 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:39.247321 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.263065 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.316991 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.335546 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.415861 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:39.684377 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.723637 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.734525 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.742910 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.816159 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.845695 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.851894 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.909904 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.919519 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:39.958645 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:40.002885 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.060803 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.083119 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.083130 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.091603 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.116534 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.118647 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.153334 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.165946 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.171921 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
W20240503 02:13:40.392024 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:40.415796 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.463742 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.822135 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.888988 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.933080 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:40.934996 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:41.004338 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:41.042423 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:41.058444 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:41.121620 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:41.143038 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:41.170423 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:41.264847 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:41.281666 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:41.340690 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:41.601730 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:41.742470 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:41.762414 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:41.880422 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:41.920760 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.097213 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.149017 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.165694 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:42.166937 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.199172 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.291026 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.322810 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.323711 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.396920 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.397614 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.425432 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.458596 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:42.458972 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.462546 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.482267 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:13:42.495842 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.564787 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:13:42.565779 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.580821 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.591811 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.626000 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:13:42.626996 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
W20240503 02:13:42.627107 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:13:42.665539 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:13:42.666611 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.675444 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:13:42.676405 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:42.775141 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.778931 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:42.893592 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.113535 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:13:43.154165 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:43.334240 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.363116 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.436240 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.491992 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.542801 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.556589 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:43.588208 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.610567 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.768474 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.836575 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.844277 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.910296 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:43.975478 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:44.163029 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.195489 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.223105 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.375604 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.477553 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:44.548511 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.549127 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.583073 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.652875 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.669342 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.677336 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.732767 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.766269 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.795745 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.798070 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.846141 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.917615 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:44.945562 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:44.961661 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.092793 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.169015 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.190651 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
W20240503 02:13:45.247651 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.331424 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.335678 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.357075 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.441267 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:45.469789 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.727262 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.870515 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:45.888602 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.013303 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:46.056046 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.079089 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.098608 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.114089 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.215549 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.296592 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.311580 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.377980 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:46.389595 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:46.508883 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:46.547072 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:46.718379 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:46.792428 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:46.899303 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:46.998587 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.018711 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.027040 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.033586 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.126907 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.138389 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.205296 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.313207 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:47.326292 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.438614 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.480427 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.518355 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:13:47.530573 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.545325 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.554376 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.633350 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 02:13:47.681751 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:13:47.682791 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.693578 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:13:47.696300 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:13:47.713099 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:13:47.714107 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
W20240503 02:13:47.731550 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.738523 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.784010 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:47.804368 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.843088 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:47.888010 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.083444 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.196244 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.197305 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:13:48.198236 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.270128 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:48.299760 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.607452 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.630856 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.638713 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.715751 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.718232 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.730137 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.778358 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:48.890054 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:48.892163 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:49.027505 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.097507 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.108503 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.326925 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.333041 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 4 similar messages]
W20240503 02:13:49.334414 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:49.348695 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:49.451712 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.561408 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.566644 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.664558 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.685631 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.693002 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.712848 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.713371 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.738301 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.770004 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:49.815768 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.916019 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.983318 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:49.997084 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 31: this message will repeat every 5th retry.
W20240503 02:13:50.008653 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.090153 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.156744 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:50.200251 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.265065 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.279872 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.286101 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.339148 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.524818 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.612665 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.662876 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:50.671556 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:50.688944 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:50.980281 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.074847 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:51.092717 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.101294 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.163806 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.183904 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.270695 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.284376 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.313416 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.457355 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:51.477432 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:51.581575 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:51.587919 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:51.687147 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:51.763110 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:51.838627 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:51.851060 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:52.007709 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.083858 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.113339 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.245879 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.247892 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.268600 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.316481 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.317152 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.402045 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:52.438812 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.486593 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.504994 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.617798 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.667968 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 36: this message will repeat every 5th retry.
W20240503 02:13:52.676517 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:13:52.677502 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 02:13:52.728477 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:13:52.730149 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.747063 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:13:52.758755 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:13:52.765046 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.795742 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.820056 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:13:52.843314 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.855696 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:52.858736 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.891633 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.930634 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:52.949419 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.212826 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.213366 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:13:53.358732 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:53.452205 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.583504 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.613137 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.778467 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:53.801733 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.810339 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.850617 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.872803 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:53.916678 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:54.107822 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.118494 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:54.118647 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.249218 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.275642 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.372251 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:54.499490 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.557739 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.615742 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.646682 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.736794 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.842617 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:54.853963 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.891165 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.904911 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.911834 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:54.970141 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.010893 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 41: this message will repeat every 5th retry.
W20240503 02:13:55.023844 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.121109 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.173441 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.194484 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.216589 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.264135 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.280577 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.301329 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:55.306383 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.438338 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.462921 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.549408 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.796767 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:55.823531 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:55.933394 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.108750 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.218947 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:56.269747 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.284246 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.304622 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:56.331729 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.385231 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.397306 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.472931 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.521062 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:56.571744 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:56.638902 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:56.688259 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:56.834475 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:56.843447 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:56.844138 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:56.972535 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.082649 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:57.232913 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.235991 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.318815 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.319324 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.339705 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.351579 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 46: this message will repeat every 5th retry.
W20240503 02:13:57.358135 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.494220 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.521716 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.566082 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.569582 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.611686 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:57.619786 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.696575 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:13:57.697676 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.755685 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:13:57.756711 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:13:57.756989 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.785907 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:13:57.786994 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.812654 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.890605 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:13:57.899147 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:57.915251 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.058233 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:58.077548 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:13:58.078585 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.176566 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.401016 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:13:58.517426 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.550549 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:58.675269 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.796957 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:58.861065 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.942973 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.956012 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.956293 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.971577 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.983251 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:58.996615 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:59.000927 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.036597 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:59.187784 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.379194 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.471093 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.478569 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.483500 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.483532 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.618917 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:13:59.674732 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.759084 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.764488 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.801244 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.825517 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:13:59.835492 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 51: this message will repeat every 5th retry.
W20240503 02:13:59.906615 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.003523 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:00.012946 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.020603 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.021216 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.118000 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.149323 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.207924 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.224923 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.328670 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.336820 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.415946 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.473157 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:00.565809 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:00.719460 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.062726 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:01.075356 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.203792 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.280351 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.372880 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:01.408751 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.425407 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:01.475158 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.497325 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.520998 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.589165 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:01.621291 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.643294 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:01.669523 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:01.684585 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:01.781234 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:01.947939 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.073984 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.115875 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.200975 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:02.206321 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.208968 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.245039 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.318013 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 56: this message will repeat every 5th retry.
W20240503 02:14:02.337656 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.349470 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.401937 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.475436 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.485616 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.618903 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.695688 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.740487 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.741345 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:14:02.780476 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:02.782150 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:14:02.783573 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:02.786419 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.795552 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.803306 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:02.804734 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.931126 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:02.977595 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.010919 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.017251 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:03.101198 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:14:03.102062 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.342609 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:03.622959 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:03.649982 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:03.664513 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.718199 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.762384 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:03.817368 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.871834 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.874899 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.880959 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:03.895325 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:03.969790 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:03.999850 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:04.045010 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:04.067188 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.287806 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.407495 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:04.460813 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:04.498332 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.531231 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.532038 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.545845 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.632323 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.769251 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.845638 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:04.864032 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.934916 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.942282 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 61: this message will repeat every 5th retry.
W20240503 02:14:04.995713 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:04.997720 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.001192 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.086823 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.087327 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.173774 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.188530 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.190990 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.262187 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.331964 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.419314 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:05.503072 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.679205 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.688809 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:05.722615 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:14:05.723799 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:05.935757 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:05.984349 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.081017 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:06.183909 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.188221 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.188349 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:06.372572 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.384658 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.409976 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.416206 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.431787 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:06.435437 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.449221 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:06.521924 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:06.538661 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:06.835034 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:06.862740 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:06.881385 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:06.965497 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.054564 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.098604 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:07.166775 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.215585 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.328963 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.343859 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.361150 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.362794 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:07.400532 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.477771 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 66: this message will repeat every 5th retry.
W20240503 02:14:07.549814 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.605791 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.634160 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.691533 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.735419 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.745887 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.784760 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.786707 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 02:14:07.787680 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.854478 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:07.855288 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:14:07.855950 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:07.869587 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:07.886462 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:14:07.889343 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.118393 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:08.170045 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.232815 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.317126 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:14:08.319561 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:08.372061 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.504591 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.609468 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.679174 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.703706 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:08.842923 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:08.845527 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:08.854352 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:08.862587 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.862692 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:08.907336 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.024263 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.068101 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:09.113469 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:09.146174 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:09.193980 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.283447 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.312270 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:09.400316 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.537338 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.732185 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.756693 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.851344 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.866138 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:09.923974 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.936713 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 71: this message will repeat every 5th retry.
W20240503 02:14:09.939394 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.967540 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:09.967908 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.051043 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.169353 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.171497 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.231329 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.238003 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.302306 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.308406 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.348764 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:10.390489 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.487438 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.611935 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.702447 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.895452 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:10.950349 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:11.062597 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:11.222038 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:11.287201 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:11.317008 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:11.397025 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:11.422813 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:11.470125 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:11.474411 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:11.495051 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:11.512195 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:11.520471 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:11.632757 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:11.651463 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:11.694538 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:11.715328 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
I20240503 02:14:11.790695 9941 maintenance_manager.cc:419] P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033385
W20240503 02:14:11.860535 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
I20240503 02:14:11.867030 9940 maintenance_manager.cc:643] P 1b81c1c3148f4f3eabfcd10eecb5c6b9: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.075s user 0.054s sys 0.015s Metrics: {"bytes_written":80543,"cfile_init":1,"dirs.queue_time_us":463,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":3246,"drs_written":1,"lbm_read_time_us":2200,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":3,"lbm_write_time_us":1579,"lbm_writes_lt_1ms":34,"peak_mem_usage":0,"rows_written":223,"thread_start_us":5737,"threads_started":2,"wal-append.queue_time_us":5716}
I20240503 02:14:11.868909 9941 maintenance_manager.cc:419] P 1b81c1c3148f4f3eabfcd10eecb5c6b9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 35147 bytes on disk
I20240503 02:14:11.869747 9940 maintenance_manager.cc:643] P 1b81c1c3148f4f3eabfcd10eecb5c6b9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 02:14:11.951750 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:12.142776 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.236701 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.347720 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.362046 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 76: this message will repeat every 5th retry.
W20240503 02:14:12.371862 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:12.374920 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.410564 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.441974 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.467572 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.539593 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.544620 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.556399 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.562431 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.586225 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.697463 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.699034 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.726969 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.737668 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:12.827632 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:14:12.871788 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 02:14:12.909727 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:14:12.946756 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:12.948621 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.048513 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:13.067575 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.139854 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:13.265434 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.420948 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:14:13.422251 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:13.428750 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.547159 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.610054 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.676420 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.749186 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.758520 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:13.815057 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:13.881369 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:13.883419 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:13.887559 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:13.940022 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.132148 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:14.137878 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:14.158759 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.175787 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.222249 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:14.307852 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:14.463270 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.478482 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:14.569038 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.629416 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.796730 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.798782 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.799140 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.811630 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:14.856659 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:14.868479 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.005273 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.053763 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.087088 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.089327 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 81: this message will repeat every 5th retry.
W20240503 02:14:15.112771 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.113272 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.137295 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.228299 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.238271 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.375324 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.435766 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:15.553768 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.603112 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.630407 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.862774 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.862733 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:15.863816 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:15.878147 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:16.074748 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:16.210726 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:16.305971 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:16.314043 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:16.324893 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:16.400560 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:16.486992 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:16.515197 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:16.545028 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:16.703167 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:16.766597 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:16.853276 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:16.863278 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:16.929436 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:16.975723 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.162923 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.163972 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:17.225334 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.233528 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.344491 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.351227 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.357276 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.401376 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.434736 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.454129 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.494271 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.508251 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.516580 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.609443 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 86: this message will repeat every 5th retry.
W20240503 02:14:17.684626 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.774211 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:17.830832 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:17.860266 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:14:17.896770 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:14:17.926127 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:14:18.017357 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.023504 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.026031 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.041265 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.125746 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:14:18.153790 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:18.251626 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.398665 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:18.454751 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:14:18.542188 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.550477 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.707583 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:18.725008 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:18.774050 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.792043 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.849912 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:18.864713 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:18.889706 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.895391 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:18.972903 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.130441 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:19.192504 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:14:19.368366 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.416422 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:19.431823 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.484655 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:19.579277 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.631101 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.661651 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:19.665132 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.716567 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.774811 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.783293 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.784415 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.847234 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:19.855511 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.874071 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.889084 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.902554 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:19.937140 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:20.163169 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:20.271025 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:20.340950 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:20.341476 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:20.357461 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:20.424659 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 91: this message will repeat every 5th retry.
W20240503 02:14:20.435716 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:20.459192 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:20.649438 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:20.800280 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:20.971417 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.008752 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.083742 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.103291 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.210986 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:21.239842 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.326758 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.357379 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:21.371511 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.415329 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:21.447186 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:14:21.513732 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(13e83ce8820b491992c041a8c21fa3df): perf score=0.033340
I20240503 02:14:21.575882 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(a2aac20a18f4444a9ff79642b23c89d9): perf score=0.033355
I20240503 02:14:21.576421 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(a2aac20a18f4444a9ff79642b23c89d9): perf score=0.033356
W20240503 02:14:21.682301 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:21.688735 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:14:21.718047 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(13e83ce8820b491992c041a8c21fa3df) complete. Timing: real 0.204s user 0.078s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":487,"dirs.queue_time_us":23677,"dirs.run_cpu_time_us":311,"dirs.run_wall_time_us":1540,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":934,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":47985,"threads_started":3,"wal-append.queue_time_us":8150}
W20240503 02:14:21.727068 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:14:21.727978 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(13e83ce8820b491992c041a8c21fa3df): 4838 bytes on disk
I20240503 02:14:21.738202 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(13e83ce8820b491992c041a8c21fa3df) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":121,"lbm_reads_lt_1ms":4}
W20240503 02:14:21.786149 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:21.799419 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:14:21.854750 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(a2aac20a18f4444a9ff79642b23c89d9) complete. Timing: real 0.256s user 0.083s sys 0.043s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":21994,"dirs.queue_time_us":47640,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":2540,"drs_written":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1204,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":82083,"threads_started":3,"wal-append.queue_time_us":13019}
I20240503 02:14:21.865303 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(390ad47a9a534abb81d59dc7604c72d3): perf score=0.033433
W20240503 02:14:21.883384 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:14:21.886267 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(a2aac20a18f4444a9ff79642b23c89d9) complete. Timing: real 0.309s user 0.070s sys 0.021s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":15052,"dirs.queue_time_us":9066,"dirs.run_cpu_time_us":339,"dirs.run_wall_time_us":6016,"drs_written":1,"lbm_read_time_us":128,"lbm_reads_lt_1ms":4,"lbm_write_time_us":9262,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":22,"peak_mem_usage":0,"rows_written":689,"thread_start_us":37307,"threads_started":3,"wal-append.queue_time_us":9510}
I20240503 02:14:21.890206 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(a2aac20a18f4444a9ff79642b23c89d9): 4923 bytes on disk
W20240503 02:14:21.894011 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:14:21.900249 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(a2aac20a18f4444a9ff79642b23c89d9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":404,"lbm_reads_lt_1ms":4}
W20240503 02:14:21.934841 10026 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.68:35025, user_credentials={real_user=slave}} blocked reactor thread for 80475.4us
W20240503 02:14:22.075918 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
I20240503 02:14:22.089999 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(390ad47a9a534abb81d59dc7604c72d3) complete. Timing: real 0.224s user 0.065s sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":51,"dirs.run_cpu_time_us":244,"dirs.run_wall_time_us":2091,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1107,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":13031,"threads_started":1,"wal-append.queue_time_us":12887}
I20240503 02:14:22.094161 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(a2aac20a18f4444a9ff79642b23c89d9): 4923 bytes on disk
W20240503 02:14:22.096029 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:14:22.097023 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(a2aac20a18f4444a9ff79642b23c89d9) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":172,"lbm_reads_lt_1ms":4}
I20240503 02:14:22.099766 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(390ad47a9a534abb81d59dc7604c72d3): 4583 bytes on disk
I20240503 02:14:22.101701 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(390ad47a9a534abb81d59dc7604c72d3) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
W20240503 02:14:22.153728 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.187386 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.198100 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.258944 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:22.274688 12973 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.69:33471, user_credentials={real_user=slave}} blocked reactor thread for 96970us
W20240503 02:14:22.275800 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.281895 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:22.292246 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.367169 12842 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.65:38119, user_credentials={real_user=slave}} blocked reactor thread for 126641us
W20240503 02:14:22.403337 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.413990 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.603607 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.605870 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.683945 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.727383 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.771696 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.827129 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 96: this message will repeat every 5th retry.
W20240503 02:14:22.840157 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:22.891356 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:22.927711 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 02:14:22.941010 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:14:23.035789 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 02:14:23.036789 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.093111 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.177320 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:23.196949 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.237735 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:23.286545 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:23.633436 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:23.634570 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.658542 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.670511 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.692188 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:23.701682 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:23.707079 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.716496 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:23.776212 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.810490 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:23.875882 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:14:23.920847 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(476f6e414dae4db3ba9a187adad75f8c): perf score=0.033353
W20240503 02:14:23.927786 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:14:23.979883 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(476f6e414dae4db3ba9a187adad75f8c) complete. Timing: real 0.059s user 0.030s sys 0.023s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":643,"dirs.run_cpu_time_us":228,"dirs.run_wall_time_us":1684,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":883,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":2141,"threads_started":2,"wal-append.queue_time_us":1766}
I20240503 02:14:23.982180 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(476f6e414dae4db3ba9a187adad75f8c): 4921 bytes on disk
I20240503 02:14:23.982889 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(476f6e414dae4db3ba9a187adad75f8c) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 02:14:24.012593 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(d2fb90eef6224ed1a1dcc57c71683bc3): perf score=0.033369
W20240503 02:14:24.051615 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:14:24.060771 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(d2fb90eef6224ed1a1dcc57c71683bc3) complete. Timing: real 0.048s user 0.035s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":498,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1708,"drs_written":1,"lbm_read_time_us":59,"lbm_reads_lt_1ms":4,"lbm_write_time_us":995,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":604,"threads_started":2,"wal-append.queue_time_us":409}
I20240503 02:14:24.063360 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(d2fb90eef6224ed1a1dcc57c71683bc3): 4838 bytes on disk
I20240503 02:14:24.064170 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(d2fb90eef6224ed1a1dcc57c71683bc3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
W20240503 02:14:24.091195 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
I20240503 02:14:24.125298 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(36457442a8e24ece9a1c2d8cdab89b78): perf score=0.033402
I20240503 02:14:24.166266 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(36457442a8e24ece9a1c2d8cdab89b78) complete. Timing: real 0.041s user 0.023s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":569,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1553,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":950,"threads_started":2,"wal-append.queue_time_us":617}
I20240503 02:14:24.168519 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(36457442a8e24ece9a1c2d8cdab89b78): 4583 bytes on disk
I20240503 02:14:24.169128 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(36457442a8e24ece9a1c2d8cdab89b78) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
I20240503 02:14:24.170805 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(d2fb90eef6224ed1a1dcc57c71683bc3): perf score=0.033411
W20240503 02:14:24.181794 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
I20240503 02:14:24.213761 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(d2fb90eef6224ed1a1dcc57c71683bc3) complete. Timing: real 0.043s user 0.010s sys 0.032s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":338,"dirs.run_cpu_time_us":257,"dirs.run_wall_time_us":1796,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":902,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":322,"threads_started":1,"wal-append.queue_time_us":458}
I20240503 02:14:24.216336 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(d2fb90eef6224ed1a1dcc57c71683bc3): 4838 bytes on disk
I20240503 02:14:24.217152 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(d2fb90eef6224ed1a1dcc57c71683bc3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4}
W20240503 02:14:24.262768 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.277086 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:24.343176 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.511097 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:24.583329 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.584594 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.678678 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:24.708526 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.809309 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.823598 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.853539 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.857204 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.895083 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:24.899489 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.934322 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:24.982395 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.039021 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.063091 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:25.074759 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.096057 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.097824 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.192857 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 101: this message will repeat every 5th retry.
W20240503 02:14:25.230494 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.402555 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.463490 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:25.503268 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.654119 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:25.967051 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:26.031821 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:26.044621 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:26.070300 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:26.231128 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:26.234216 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:26.244618 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:26.271411 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:26.272258 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:26.349210 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:26.350332 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:26.372195 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:26.528734 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:26.543510 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:26.631652 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:26.749249 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:26.888933 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:26.976680 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.095467 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.099133 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:27.130210 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.177911 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:27.182054 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.249352 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.328001 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.373689 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.429167 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.443737 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.472051 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:27.490528 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:14:27.496963 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(9f8b006a679a4b8d8d3551310b78fb3a): perf score=0.033366
I20240503 02:14:27.521379 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(2fadc7e9c7604b5d8f7b8751b6ca78a7): perf score=0.033357
I20240503 02:14:27.558606 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(9f8b006a679a4b8d8d3551310b78fb3a) complete. Timing: real 0.061s user 0.054s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":418,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":2275,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":694,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":617,"threads_started":2,"wal-append.queue_time_us":430}
I20240503 02:14:27.561681 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(9f8b006a679a4b8d8d3551310b78fb3a): 4581 bytes on disk
I20240503 02:14:27.562350 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(9f8b006a679a4b8d8d3551310b78fb3a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240503 02:14:27.563452 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:14:27.567451 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(0a5125fbc8ec4ba887abce94a84efbdc): perf score=0.033381
I20240503 02:14:27.594427 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(2fadc7e9c7604b5d8f7b8751b6ca78a7) complete. Timing: real 0.073s user 0.051s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":498,"dirs.run_cpu_time_us":219,"dirs.run_wall_time_us":2266,"drs_written":1,"lbm_read_time_us":87,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1104,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":707,"threads_started":2,"wal-append.queue_time_us":479}
I20240503 02:14:27.597415 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(2fadc7e9c7604b5d8f7b8751b6ca78a7): 4836 bytes on disk
I20240503 02:14:27.598127 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(2fadc7e9c7604b5d8f7b8751b6ca78a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":102,"lbm_reads_lt_1ms":4}
W20240503 02:14:27.598376 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:14:27.599119 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(0a5125fbc8ec4ba887abce94a84efbdc): perf score=0.033395
W20240503 02:14:27.601179 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.603690 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:14:27.613855 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(0a5125fbc8ec4ba887abce94a84efbdc) complete. Timing: real 0.046s user 0.036s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":211,"dirs.run_wall_time_us":2097,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":839,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":309,"threads_started":1,"wal-append.queue_time_us":448}
I20240503 02:14:27.616504 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(0a5125fbc8ec4ba887abce94a84efbdc): 4923 bytes on disk
I20240503 02:14:27.617094 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(0a5125fbc8ec4ba887abce94a84efbdc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":74,"lbm_reads_lt_1ms":4}
W20240503 02:14:27.636930 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:14:27.674612 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(0a5125fbc8ec4ba887abce94a84efbdc) complete. Timing: real 0.075s user 0.049s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":465,"dirs.run_cpu_time_us":200,"dirs.run_wall_time_us":1703,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":792,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":547,"threads_started":2,"wal-append.queue_time_us":349}
I20240503 02:14:27.677094 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(0a5125fbc8ec4ba887abce94a84efbdc): 4923 bytes on disk
I20240503 02:14:27.677745 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(0a5125fbc8ec4ba887abce94a84efbdc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4}
W20240503 02:14:27.742086 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 106: this message will repeat every 5th retry.
W20240503 02:14:27.867947 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:27.973950 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:14:28.008046 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.016877 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.043963 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:14:28.059867 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:28.066872 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:14:28.246356 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:28.247485 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.319764 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:28.324600 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:28.326243 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.470585 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:28.524750 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.528225 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.610539 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.665275 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:28.666391 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.892187 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.894114 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:28.898134 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:28.987340 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.015714 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.024331 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:29.050191 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:29.210973 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.225662 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.450359 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:29.465667 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.553197 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:29.570016 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:29.576092 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.588984 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.598341 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.689512 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.691109 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.734432 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.770413 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:29.802256 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:14:29.867089 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(5547b33324db46c886287b4ee5035e43): perf score=0.033371
I20240503 02:14:29.886780 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(31f72bc850c74289aa3a59ecc3dcc287): perf score=0.033364
I20240503 02:14:29.908592 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(5547b33324db46c886287b4ee5035e43) complete. Timing: real 0.041s user 0.029s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":636,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":1593,"drs_written":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4,"lbm_write_time_us":715,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":847,"threads_started":2,"wal-append.queue_time_us":450}
I20240503 02:14:29.910792 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(5547b33324db46c886287b4ee5035e43): 4583 bytes on disk
I20240503 02:14:29.911931 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(5547b33324db46c886287b4ee5035e43) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 02:14:29.941448 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(31f72bc850c74289aa3a59ecc3dcc287) complete. Timing: real 0.054s user 0.040s sys 0.007s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":3062,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":1951,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":834,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":3231,"threads_started":2,"wal-append.queue_time_us":370}
W20240503 02:14:29.942838 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:14:29.944290 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(31f72bc850c74289aa3a59ecc3dcc287): 4838 bytes on disk
W20240503 02:14:29.944820 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:14:29.945523 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(31f72bc850c74289aa3a59ecc3dcc287) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240503 02:14:29.946306 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:14:29.947993 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(da288f76df0a409eb01ee431d3b83b49): perf score=0.033395
W20240503 02:14:29.969786 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
I20240503 02:14:29.994019 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(da288f76df0a409eb01ee431d3b83b49) complete. Timing: real 0.046s user 0.040s sys 0.003s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":421,"dirs.run_cpu_time_us":252,"dirs.run_wall_time_us":1517,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":770,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":533,"threads_started":2,"wal-append.queue_time_us":357}
I20240503 02:14:29.996708 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(da288f76df0a409eb01ee431d3b83b49): 4923 bytes on disk
I20240503 02:14:29.997493 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(da288f76df0a409eb01ee431d3b83b49) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
I20240503 02:14:29.999287 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(31f72bc850c74289aa3a59ecc3dcc287): perf score=0.033396
W20240503 02:14:30.007392 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
I20240503 02:14:30.049430 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(31f72bc850c74289aa3a59ecc3dcc287) complete. Timing: real 0.050s user 0.032s sys 0.016s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":58,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":1604,"drs_written":1,"lbm_read_time_us":124,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1479,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":333,"threads_started":1,"wal-append.queue_time_us":460}
I20240503 02:14:30.051990 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(31f72bc850c74289aa3a59ecc3dcc287): 4838 bytes on disk
I20240503 02:14:30.052778 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(31f72bc850c74289aa3a59ecc3dcc287) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
W20240503 02:14:30.101830 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.109673 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 111: this message will repeat every 5th retry.
W20240503 02:14:30.122853 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.169153 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.175490 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.314756 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.390643 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:30.529179 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.687917 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.767688 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.834182 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:30.939067 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:30.958647 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:31.038102 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:31.113297 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:31.250036 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:31.267374 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:31.319128 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:31.327252 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:31.480203 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:31.600421 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:31.647393 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:31.667788 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:31.681577 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:31.685271 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:31.746673 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:31.932104 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:31.969065 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.005604 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.047781 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.061074 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.073930 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.083393 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:32.157488 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.177399 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.279060 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.286278 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:32.363730 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.367384 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.381232 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.473137 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:32.499419 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 116: this message will repeat every 5th retry.
W20240503 02:14:32.532399 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.651105 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.730769 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.733777 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.756480 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:32.866186 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:32.985884 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:32.996286 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:33.075520 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:14:33.095502 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:14:33.188288 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:33.198563 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:33.359683 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:33.437891 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:33.438949 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:14:33.442806 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(206e6a3850ff4242b30fa10c3f751fd2): perf score=0.033355
I20240503 02:14:33.477730 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(0dc2733b737b431684ce439c3d3fa762): perf score=0.033353
W20240503 02:14:33.520359 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:14:33.522078 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(0dc2733b737b431684ce439c3d3fa762) complete. Timing: real 0.044s user 0.021s sys 0.019s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":614,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":2086,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":926,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1408,"threads_started":2,"wal-append.queue_time_us":1068}
I20240503 02:14:33.522079 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(206e6a3850ff4242b30fa10c3f751fd2) complete. Timing: real 0.079s user 0.056s sys 0.019s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":443,"dirs.run_cpu_time_us":332,"dirs.run_wall_time_us":1985,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":10832,"threads_started":2,"wal-append.queue_time_us":10665}
I20240503 02:14:33.524773 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(0dc2733b737b431684ce439c3d3fa762): 4581 bytes on disk
I20240503 02:14:33.524816 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(206e6a3850ff4242b30fa10c3f751fd2): 4838 bytes on disk
I20240503 02:14:33.525537 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(0dc2733b737b431684ce439c3d3fa762) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":131,"lbm_reads_lt_1ms":4}
I20240503 02:14:33.525547 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(206e6a3850ff4242b30fa10c3f751fd2) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240503 02:14:33.531433 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:33.567456 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:33.567647 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:14:33.583171 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(af7beeafa08f477796e5070fbc177dcc): perf score=0.033388
W20240503 02:14:33.585777 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:14:33.623725 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(af7beeafa08f477796e5070fbc177dcc) complete. Timing: real 0.040s user 0.026s sys 0.010s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":440,"dirs.run_cpu_time_us":254,"dirs.run_wall_time_us":1524,"drs_written":1,"lbm_read_time_us":67,"lbm_reads_lt_1ms":4,"lbm_write_time_us":845,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":584,"threads_started":2,"wal-append.queue_time_us":402}
I20240503 02:14:33.626246 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(0dc2733b737b431684ce439c3d3fa762): perf score=0.033398
W20240503 02:14:33.667398 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:33.670068 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
I20240503 02:14:33.672317 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(0dc2733b737b431684ce439c3d3fa762) complete. Timing: real 0.046s user 0.032s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":74,"dirs.run_cpu_time_us":192,"dirs.run_wall_time_us":1575,"drs_written":1,"lbm_read_time_us":73,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1004,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4414,"threads_started":1,"wal-append.queue_time_us":4365}
I20240503 02:14:33.675017 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(af7beeafa08f477796e5070fbc177dcc): 4923 bytes on disk
I20240503 02:14:33.675752 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(af7beeafa08f477796e5070fbc177dcc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
I20240503 02:14:33.677814 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(0dc2733b737b431684ce439c3d3fa762): 4581 bytes on disk
I20240503 02:14:33.678417 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(0dc2733b737b431684ce439c3d3fa762) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 02:14:33.701543 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:33.906517 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:33.938962 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:33.950976 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:33.998332 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.040135 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:14:34.046782 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:34.160311 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.192106 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:34.424214 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:34.427922 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.464532 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.481593 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.503671 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.525105 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:34.538568 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.560626 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.561897 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.582726 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.642064 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.721436 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.760701 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:34.762553 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:34.832055 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.894745 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 121: this message will repeat every 5th retry.
W20240503 02:14:34.955003 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:34.979709 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:35.057020 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:35.088683 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:35.181936 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:35.235050 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:35.374895 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:35.593464 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:35.638463 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:35.660800 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:35.698794 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
I20240503 02:14:35.699208 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(20366a3e8dcc4996bfdd1d60ef1be077): perf score=0.033337
I20240503 02:14:35.763451 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(20366a3e8dcc4996bfdd1d60ef1be077) complete. Timing: real 0.064s user 0.052s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":521,"dirs.run_cpu_time_us":279,"dirs.run_wall_time_us":1921,"drs_written":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4,"lbm_write_time_us":751,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":827,"threads_started":2,"wal-append.queue_time_us":579}
I20240503 02:14:35.766429 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(20366a3e8dcc4996bfdd1d60ef1be077): 4923 bytes on disk
I20240503 02:14:35.767156 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(20366a3e8dcc4996bfdd1d60ef1be077) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20240503 02:14:35.802588 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(fc0a049eeb0047558d3c9eb370467c53): perf score=0.033360
I20240503 02:14:35.803918 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(73ae9bc7a96149259bd807369c2daacd): perf score=0.033370
I20240503 02:14:35.867621 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(73ae9bc7a96149259bd807369c2daacd) complete. Timing: real 0.063s user 0.031s sys 0.015s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":445,"dirs.run_cpu_time_us":285,"dirs.run_wall_time_us":1789,"drs_written":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4,"lbm_write_time_us":811,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1234,"threads_started":2,"wal-append.queue_time_us":569}
I20240503 02:14:35.871644 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(73ae9bc7a96149259bd807369c2daacd): 4838 bytes on disk
I20240503 02:14:35.873220 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(73ae9bc7a96149259bd807369c2daacd) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":138,"lbm_reads_lt_1ms":4}
I20240503 02:14:35.880038 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(fc0a049eeb0047558d3c9eb370467c53) complete. Timing: real 0.077s user 0.020s sys 0.030s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":471,"dirs.run_cpu_time_us":212,"dirs.run_wall_time_us":3528,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":851,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5932,"threads_started":2,"wal-append.queue_time_us":5696}
I20240503 02:14:35.882642 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(fc0a049eeb0047558d3c9eb370467c53): 4583 bytes on disk
I20240503 02:14:35.883385 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(fc0a049eeb0047558d3c9eb370467c53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
W20240503 02:14:36.025897 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:36.054428 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:36.166519 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:36.195606 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:36.258031 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:36.286260 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:36.290498 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:36.335398 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:36.351199 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:36.371383 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:36.444962 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:36.465462 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:36.514938 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:36.566481 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:36.793198 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:36.865825 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:36.910229 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:36.965955 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.037299 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.093312 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.103909 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:37.110369 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.134765 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.159420 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:37.175733 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:37.219921 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.228494 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.357369 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.364820 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 126: this message will repeat every 5th retry.
W20240503 02:14:37.542496 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.549603 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.557745 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.572064 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.669116 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:37.690536 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.735504 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:37.993057 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 46 similar messages]
W20240503 02:14:37.995347 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:38.045118 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:38.121903 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:38.122658 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:14:38.128127 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:14:38.140720 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:38.201759 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:38.384533 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:14:38.405706 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(12da2248f416404da29a4557b90daef7): perf score=0.033347
W20240503 02:14:38.420110 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:38.422035 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
I20240503 02:14:38.474455 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(12da2248f416404da29a4557b90daef7) complete. Timing: real 0.068s user 0.053s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":953,"dirs.run_cpu_time_us":226,"dirs.run_wall_time_us":2110,"drs_written":1,"lbm_read_time_us":61,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1085,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1551,"threads_started":2,"wal-append.queue_time_us":906}
I20240503 02:14:38.476699 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(12da2248f416404da29a4557b90daef7): 4838 bytes on disk
I20240503 02:14:38.477586 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(12da2248f416404da29a4557b90daef7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
W20240503 02:14:38.499022 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:38.513315 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:14:38.514658 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:14:38.541850 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(5aa7ed62ebf44431a1a6db26ef73052a): perf score=0.033378
W20240503 02:14:38.566444 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:14:38.582188 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(5aa7ed62ebf44431a1a6db26ef73052a) complete. Timing: real 0.040s user 0.024s sys 0.014s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":622,"dirs.run_cpu_time_us":346,"dirs.run_wall_time_us":1893,"drs_written":1,"lbm_read_time_us":54,"lbm_reads_lt_1ms":4,"lbm_write_time_us":926,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":694,"threads_started":2,"wal-append.queue_time_us":399}
I20240503 02:14:38.584602 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(be61de73e08c4998a51f02cafa46d421): perf score=0.033380
I20240503 02:14:38.647473 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(be61de73e08c4998a51f02cafa46d421): perf score=0.033401
I20240503 02:14:38.652175 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(be61de73e08c4998a51f02cafa46d421) complete. Timing: real 0.067s user 0.037s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":901,"dirs.run_cpu_time_us":268,"dirs.run_wall_time_us":1889,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":796,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":306,"threads_started":1,"wal-append.queue_time_us":442}
I20240503 02:14:38.654724 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(5aa7ed62ebf44431a1a6db26ef73052a): 4923 bytes on disk
I20240503 02:14:38.655521 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(5aa7ed62ebf44431a1a6db26ef73052a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20240503 02:14:38.658385 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(be61de73e08c4998a51f02cafa46d421): 4583 bytes on disk
I20240503 02:14:38.658970 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(be61de73e08c4998a51f02cafa46d421) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
I20240503 02:14:38.688692 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(be61de73e08c4998a51f02cafa46d421) complete. Timing: real 0.041s user 0.015s sys 0.021s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":382,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1730,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":783,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":509,"threads_started":2,"wal-append.queue_time_us":340}
I20240503 02:14:38.690840 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(be61de73e08c4998a51f02cafa46d421): 4583 bytes on disk
I20240503 02:14:38.691422 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(be61de73e08c4998a51f02cafa46d421) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":77,"lbm_reads_lt_1ms":4}
W20240503 02:14:38.735090 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:38.884093 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:38.897414 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:38.946708 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:38.966398 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:38.992756 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.009240 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.027190 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:39.046042 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:39.115345 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:39.116343 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:39.138553 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.223474 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:39.290438 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.293742 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.399530 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.418438 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:39.420639 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.551566 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.557956 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:39.633632 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.719380 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.736029 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.741859 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.840237 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:39.841107 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.848958 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:39.877565 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 131: this message will repeat every 5th retry.
W20240503 02:14:39.886521 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:39.927110 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:39.932031 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.061681 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.110975 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:40.161773 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.183907 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.239331 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.467319 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.711350 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:40.725936 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.903580 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.922277 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:40.927738 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:40.972302 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
I20240503 02:14:40.999047 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(1fa4eb577d7f42fd8e4a9a4391a0546f): perf score=0.033343
I20240503 02:14:41.040328 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(1fa4eb577d7f42fd8e4a9a4391a0546f) complete. Timing: real 0.041s user 0.022s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1365,"dirs.run_cpu_time_us":248,"dirs.run_wall_time_us":3168,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":976,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1485,"threads_started":2,"wal-append.queue_time_us":369}
I20240503 02:14:41.042471 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(1fa4eb577d7f42fd8e4a9a4391a0546f): 4583 bytes on disk
I20240503 02:14:41.043156 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(1fa4eb577d7f42fd8e4a9a4391a0546f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 02:14:41.045413 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(ee635c5c4d2c4befa0283cb8226f28d7): perf score=0.033352
I20240503 02:14:41.120422 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(ee635c5c4d2c4befa0283cb8226f28d7) complete. Timing: real 0.075s user 0.062s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":67,"dirs.run_cpu_time_us":284,"dirs.run_wall_time_us":2025,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":726,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":283,"threads_started":1,"wal-append.queue_time_us":490}
I20240503 02:14:41.122678 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(ee635c5c4d2c4befa0283cb8226f28d7): 4923 bytes on disk
I20240503 02:14:41.123368 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(ee635c5c4d2c4befa0283cb8226f28d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":122,"lbm_reads_lt_1ms":4}
W20240503 02:14:41.165319 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:14:41.179988 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(35aea508f59f4348b05d93a5afdde4ba): perf score=0.033381
W20240503 02:14:41.186646 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:14:41.212728 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(1fa4eb577d7f42fd8e4a9a4391a0546f): perf score=0.033402
I20240503 02:14:41.219871 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(35aea508f59f4348b05d93a5afdde4ba) complete. Timing: real 0.040s user 0.025s sys 0.012s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":363,"dirs.run_cpu_time_us":220,"dirs.run_wall_time_us":1902,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":679,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":525,"threads_started":2,"wal-append.queue_time_us":386}
I20240503 02:14:41.222574 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(35aea508f59f4348b05d93a5afdde4ba): 4836 bytes on disk
I20240503 02:14:41.223294 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(35aea508f59f4348b05d93a5afdde4ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20240503 02:14:41.251901 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(1fa4eb577d7f42fd8e4a9a4391a0546f) complete. Timing: real 0.039s user 0.015s sys 0.020s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":430,"dirs.run_cpu_time_us":210,"dirs.run_wall_time_us":1642,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":671,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":556,"threads_started":2,"wal-append.queue_time_us":337}
I20240503 02:14:41.253986 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(1fa4eb577d7f42fd8e4a9a4391a0546f): 4583 bytes on disk
I20240503 02:14:41.254503 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(1fa4eb577d7f42fd8e4a9a4391a0546f) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":66,"lbm_reads_lt_1ms":4}
W20240503 02:14:41.291620 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:41.338037 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:41.346807 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:41.360986 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:41.431813 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:41.518074 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:41.529526 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:41.550472 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:41.562237 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:41.588065 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:41.600889 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:41.744230 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:41.803702 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:41.839207 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:41.878554 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:41.905346 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:41.908869 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.032227 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.085299 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:42.157867 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.172618 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:42.233543 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 136: this message will repeat every 5th retry.
W20240503 02:14:42.276615 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.364173 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.387183 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.390309 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.435978 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.464424 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.490475 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:42.598047 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.632943 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.799571 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:42.942126 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:42.971709 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:43.042025 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:43.064240 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:43.142493 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:14:43.291903 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:43.329887 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:43.333432 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:14:43.384228 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:14:43.393599 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(e94a48cad1d648b18d709b46566d810b): perf score=0.033339
W20240503 02:14:43.450477 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 02:14:43.453248 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(e94a48cad1d648b18d709b46566d810b) complete. Timing: real 0.059s user 0.038s sys 0.014s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":6423,"dirs.run_cpu_time_us":297,"dirs.run_wall_time_us":1827,"drs_written":1,"lbm_read_time_us":75,"lbm_reads_lt_1ms":4,"lbm_write_time_us":858,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":7706,"threads_started":2,"wal-append.queue_time_us":1252}
I20240503 02:14:43.455849 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(e94a48cad1d648b18d709b46566d810b): 4839 bytes on disk
I20240503 02:14:43.456405 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(e94a48cad1d648b18d709b46566d810b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4}
I20240503 02:14:43.493574 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(674ea54f2cbd4c438cab5a7723c5a2c6): perf score=0.033364
I20240503 02:14:43.521880 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(674ea54f2cbd4c438cab5a7723c5a2c6): perf score=0.033366
I20240503 02:14:43.541291 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(674ea54f2cbd4c438cab5a7723c5a2c6) complete. Timing: real 0.047s user 0.037s sys 0.008s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":460,"dirs.run_cpu_time_us":260,"dirs.run_wall_time_us":1812,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":711,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":644,"threads_started":2,"wal-append.queue_time_us":445}
I20240503 02:14:43.543792 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(ceb177d6952e4c978c8be93b5b026952): perf score=0.033376
W20240503 02:14:43.555002 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 02:14:43.570824 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(674ea54f2cbd4c438cab5a7723c5a2c6) complete. Timing: real 0.049s user 0.029s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":373,"dirs.run_cpu_time_us":202,"dirs.run_wall_time_us":4105,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":876,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":749,"threads_started":2,"wal-append.queue_time_us":609}
I20240503 02:14:43.573261 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(674ea54f2cbd4c438cab5a7723c5a2c6): 4923 bytes on disk
I20240503 02:14:43.573915 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(674ea54f2cbd4c438cab5a7723c5a2c6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
W20240503 02:14:43.576650 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
I20240503 02:14:43.608258 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(ceb177d6952e4c978c8be93b5b026952) complete. Timing: real 0.064s user 0.048s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":55,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":1716,"drs_written":1,"lbm_read_time_us":52,"lbm_reads_lt_1ms":4,"lbm_write_time_us":756,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":217,"threads_started":1,"wal-append.queue_time_us":322}
I20240503 02:14:43.610635 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(674ea54f2cbd4c438cab5a7723c5a2c6): 4923 bytes on disk
I20240503 02:14:43.611256 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(674ea54f2cbd4c438cab5a7723c5a2c6) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
I20240503 02:14:43.613101 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(ceb177d6952e4c978c8be93b5b026952): 4583 bytes on disk
I20240503 02:14:43.613523 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(ceb177d6952e4c978c8be93b5b026952) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4}
W20240503 02:14:43.633766 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:43.649228 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:43.660714 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:43.710971 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:43.736881 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:43.776412 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:43.901718 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:43.902813 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:43.906257 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:43.951099 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.008649 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.094357 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:44.100112 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:44.123262 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:14:44.219358 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.225737 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:44.316134 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.332594 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:44.511610 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.521391 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.564493 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:44.566639 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.583803 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.625763 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:44.788643 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.800684 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.816123 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:44.888656 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.890730 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 141: this message will repeat every 5th retry.
W20240503 02:14:44.898988 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.909435 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:44.920795 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:44.977764 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:45.054198 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:45.188498 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:45.225087 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:45.324293 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:45.347832 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:45.473317 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:45.750944 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:14:45.841836 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling FlushMRSOp(5c762709f66f44b98089e69cbace1570): perf score=0.033351
W20240503 02:14:45.878476 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:14:45.883190 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling FlushMRSOp(731649c884c3461ab562590f012727c5): perf score=0.033360
I20240503 02:14:45.910928 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: FlushMRSOp(5c762709f66f44b98089e69cbace1570) complete. Timing: real 0.069s user 0.023s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1614,"dirs.run_cpu_time_us":207,"dirs.run_wall_time_us":1651,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":797,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":4524,"threads_started":2,"wal-append.queue_time_us":456}
I20240503 02:14:45.914153 10538 maintenance_manager.cc:419] P c6ec2f1283994a3089741715f811c850: Scheduling UndoDeltaBlockGCOp(5c762709f66f44b98089e69cbace1570): 4583 bytes on disk
I20240503 02:14:45.914948 10428 maintenance_manager.cc:643] P c6ec2f1283994a3089741715f811c850: UndoDeltaBlockGCOp(5c762709f66f44b98089e69cbace1570) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4}
I20240503 02:14:45.936823 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: FlushMRSOp(731649c884c3461ab562590f012727c5) complete. Timing: real 0.053s user 0.037s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":854,"dirs.run_cpu_time_us":267,"dirs.run_wall_time_us":2181,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1328,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":2241,"threads_started":2,"wal-append.queue_time_us":429}
I20240503 02:14:45.939500 10274 maintenance_manager.cc:419] P eff8e8d11a4447218741bb994aa41082: Scheduling UndoDeltaBlockGCOp(731649c884c3461ab562590f012727c5): 4838 bytes on disk
I20240503 02:14:45.940073 10163 maintenance_manager.cc:643] P eff8e8d11a4447218741bb994aa41082: UndoDeltaBlockGCOp(731649c884c3461ab562590f012727c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4}
I20240503 02:14:45.977813 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling FlushMRSOp(49ce473013a942698081c389ace54b32): perf score=0.033389
W20240503 02:14:45.981263 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:46.023095 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
I20240503 02:14:46.037178 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: FlushMRSOp(49ce473013a942698081c389ace54b32) complete. Timing: real 0.059s user 0.057s sys 0.000s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":380,"dirs.run_cpu_time_us":269,"dirs.run_wall_time_us":1875,"drs_written":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":541,"threads_started":2,"wal-append.queue_time_us":394}
I20240503 02:14:46.039693 10141 maintenance_manager.cc:419] P 031a0afb01b6469299fa829fa3467ef7: Scheduling UndoDeltaBlockGCOp(49ce473013a942698081c389ace54b32): 4921 bytes on disk
I20240503 02:14:46.040340 10030 maintenance_manager.cc:643] P 031a0afb01b6469299fa829fa3467ef7: UndoDeltaBlockGCOp(49ce473013a942698081c389ace54b32) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4}
W20240503 02:14:46.056007 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.167110 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:46.176334 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.200670 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.335470 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.348407 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.363544 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:46.421810 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:46.472383 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.482653 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.524803 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.540010 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:46.603582 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:46.644188 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:46.790573 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:46.864315 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:46.877180 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.091972 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.120604 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:47.131933 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.168365 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:47.185012 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:47.188041 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.197111 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:47.224476 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.299466 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 146: this message will repeat every 5th retry.
W20240503 02:14:47.335574 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.449822 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.509722 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.517450 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.672183 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:47.759608 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.798334 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.847893 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.930107 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:47.957175 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:48.029630 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:48.055949 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:14:48.056027 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:48.185092 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:48.235045 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:14:48.321439 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:48.354351 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:48.409840 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:14:48.516188 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:48.644980 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:48.656083 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:48.656805 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:48.737298 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:14:48.783018 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:48.789888 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:48.837708 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:48.846141 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:48.887367 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:48.965422 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:48.981213 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.038185 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.088842 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:49.162585 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:49.163820 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:49.195742 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.224056 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:14:49.225046 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:49.248939 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.373625 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.441795 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:49.591298 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:49.659844 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:49.697425 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:49.760494 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.809002 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 151: this message will repeat every 5th retry.
W20240503 02:14:49.833699 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.891345 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.905207 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:49.918454 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:14:49.995352 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.074438 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.146368 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.237497 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.274171 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.280187 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:50.366009 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.376331 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.411329 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.598631 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:50.751555 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:50.900332 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:51.009306 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.011495 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.057597 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:51.155977 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.162835 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:51.224210 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:51.253785 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:51.370383 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.397259 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.428776 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.461565 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.464828 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:51.465881 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.572300 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.613484 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:51.639568 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:51.753623 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:51.828269 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.878439 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:51.900066 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:51.926023 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:52.074215 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:52.121438 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.135226 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.280961 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.305264 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.328514 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:52.367983 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.370114 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:52.384577 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 156: this message will repeat every 5th retry.
W20240503 02:14:52.682487 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.726994 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.744652 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.766359 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:52.772287 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:52.903743 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.012184 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.237113 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:53.237169 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 02:14:53.251955 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.288955 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.320710 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.353705 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:53.356194 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.426136 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:14:53.664341 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.665650 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:14:53.686872 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.709159 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:53.714593 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:53.739253 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:53.778179 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:53.784806 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:53.787318 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:53.831424 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:53.841349 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:14:53.842315 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:53.856590 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:53.942114 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.019618 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.085393 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:54.159456 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:54.222523 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:54.282590 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:54.285112 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.289899 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:14:54.290906 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.337082 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.348240 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:54.454607 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:54.495407 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.748054 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.761552 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:54.834347 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.865060 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:54.876266 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:54.897192 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 161: this message will repeat every 5th retry.
W20240503 02:14:55.097481 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:14:55.202404 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:55.253190 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:55.269874 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:55.314319 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:55.396548 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:55.437815 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:55.488711 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:55.700477 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:55.741922 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:14:55.748108 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(1fa4eb577d7f42fd8e4a9a4391a0546f): perf score=0.033378
W20240503 02:14:55.767745 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
I20240503 02:14:55.851500 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(1fa4eb577d7f42fd8e4a9a4391a0546f) complete. Timing: real 0.103s user 0.057s sys 0.023s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":526,"dirs.queue_time_us":8506,"dirs.run_cpu_time_us":299,"dirs.run_wall_time_us":1393,"drs_written":1,"lbm_read_time_us":130,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1182,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":9782,"threads_started":3,"wal-append.queue_time_us":782}
I20240503 02:14:55.854511 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(1fa4eb577d7f42fd8e4a9a4391a0546f): 4583 bytes on disk
I20240503 02:14:55.855790 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(1fa4eb577d7f42fd8e4a9a4391a0546f) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":159,"lbm_reads_lt_1ms":4}
W20240503 02:14:55.866060 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:55.867053 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:55.965080 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:56.006989 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:56.054234 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:56.139479 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:56.159852 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:56.163022 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:56.322222 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:56.323674 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:14:56.395121 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(36457442a8e24ece9a1c2d8cdab89b78): perf score=0.033376
W20240503 02:14:56.399068 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:56.401736 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:56.434391 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:14:56.498710 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(36457442a8e24ece9a1c2d8cdab89b78) complete. Timing: real 0.103s user 0.057s sys 0.022s Metrics: {"bytes_written":6062,"cfile_init":1,"compiler_manager_pool.queue_time_us":446,"dirs.queue_time_us":477,"dirs.run_cpu_time_us":286,"dirs.run_wall_time_us":1641,"drs_written":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1294,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1415,"threads_started":3,"wal-append.queue_time_us":496}
I20240503 02:14:56.502924 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(36457442a8e24ece9a1c2d8cdab89b78): 4583 bytes on disk
I20240503 02:14:56.506464 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(36457442a8e24ece9a1c2d8cdab89b78) complete. Timing: real 0.003s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":142,"lbm_reads_lt_1ms":4}
W20240503 02:14:56.507350 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:56.531622 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:56.596833 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:14:56.615630 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(fc0a049eeb0047558d3c9eb370467c53): perf score=0.033381
W20240503 02:14:56.653918 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
I20240503 02:14:56.684927 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(fc0a049eeb0047558d3c9eb370467c53) complete. Timing: real 0.069s user 0.053s sys 0.011s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":695,"dirs.run_cpu_time_us":282,"dirs.run_wall_time_us":2858,"drs_written":1,"lbm_read_time_us":103,"lbm_reads_lt_1ms":4,"lbm_write_time_us":797,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3889,"threads_started":2,"wal-append.queue_time_us":3448}
I20240503 02:14:56.687645 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(fc0a049eeb0047558d3c9eb370467c53): 4583 bytes on disk
I20240503 02:14:56.688388 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(fc0a049eeb0047558d3c9eb370467c53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240503 02:14:56.712220 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:56.807404 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:56.855890 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:56.866041 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:56.920912 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:56.994655 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:56.995834 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:57.002404 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:57.213459 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:57.272210 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:57.351035 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:57.379539 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:57.401161 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:57.620920 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 166: this message will repeat every 5th retry.
W20240503 02:14:57.733387 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:57.890735 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:57.925227 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:57.943321 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:57.991873 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.083068 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.097298 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.109527 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.112212 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.238397 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 02:14:58.289651 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:58.313911 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.359643 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.382033 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:58.416095 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:58.494374 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.514907 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.515635 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:58.568171 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:14:58.577312 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:58.668825 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:58.672647 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:14:58.822150 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.853964 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:58.893635 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:14:58.907789 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:58.966152 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:59.040025 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.081795 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.153553 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:59.158640 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.205137 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:59.295078 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.317754 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:14:59.318754 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.468820 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:59.483294 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.495684 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:59.506472 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:14:59.545766 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.608994 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.663761 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.686679 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.703891 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:14:59.815327 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:14:59.914477 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:00.088681 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 171: this message will repeat every 5th retry.
W20240503 02:15:00.103333 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:15:00.157152 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.211817 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.358044 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.398219 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.412653 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:00.442104 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.462728 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:15:00.539175 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(206e6a3850ff4242b30fa10c3f751fd2): perf score=0.033361
W20240503 02:15:00.571722 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.599547 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:15:00.623342 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(206e6a3850ff4242b30fa10c3f751fd2) complete. Timing: real 0.083s user 0.057s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"compiler_manager_pool.queue_time_us":1197,"dirs.queue_time_us":402,"dirs.run_cpu_time_us":224,"dirs.run_wall_time_us":1153,"drs_written":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4,"lbm_write_time_us":969,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1948,"threads_started":3,"wal-append.queue_time_us":431}
I20240503 02:15:00.625962 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(206e6a3850ff4242b30fa10c3f751fd2): 4838 bytes on disk
I20240503 02:15:00.627339 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(206e6a3850ff4242b30fa10c3f751fd2) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":100,"lbm_reads_lt_1ms":4}
W20240503 02:15:00.764925 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.773924 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.872208 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.920037 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:00.963757 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:00.980341 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:01.036648 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:01.039062 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:01.073014 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:01.136274 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:01.215716 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:15:01.224874 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(5547b33324db46c886287b4ee5035e43): perf score=0.033381
I20240503 02:15:01.274834 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(5547b33324db46c886287b4ee5035e43) complete. Timing: real 0.050s user 0.040s sys 0.007s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1179,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":1971,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":972,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5293,"threads_started":2,"wal-append.queue_time_us":4386}
I20240503 02:15:01.277523 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(731649c884c3461ab562590f012727c5): perf score=0.033357
W20240503 02:15:01.279804 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:01.340992 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(731649c884c3461ab562590f012727c5) complete. Timing: real 0.063s user 0.039s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":42,"dirs.run_cpu_time_us":293,"dirs.run_wall_time_us":1764,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":866,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":350,"threads_started":1,"wal-append.queue_time_us":486}
I20240503 02:15:01.343441 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(731649c884c3461ab562590f012727c5): 4838 bytes on disk
I20240503 02:15:01.344126 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(731649c884c3461ab562590f012727c5) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.345957 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(5547b33324db46c886287b4ee5035e43): 4583 bytes on disk
I20240503 02:15:01.347270 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(5547b33324db46c886287b4ee5035e43) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":101,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.386057 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(12da2248f416404da29a4557b90daef7): perf score=0.033349
W20240503 02:15:01.414654 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:01.447937 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(12da2248f416404da29a4557b90daef7) complete. Timing: real 0.059s user 0.045s sys 0.006s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":413,"dirs.run_cpu_time_us":265,"dirs.run_wall_time_us":1802,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":852,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":595,"threads_started":2,"wal-append.queue_time_us":447}
I20240503 02:15:01.450333 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(73ae9bc7a96149259bd807369c2daacd): perf score=0.033348
W20240503 02:15:01.453287 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:01.468777 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:01.471009 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:15:01.497630 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(73ae9bc7a96149259bd807369c2daacd) complete. Timing: real 0.047s user 0.033s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":107,"dirs.run_cpu_time_us":229,"dirs.run_wall_time_us":1522,"drs_written":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":422,"threads_started":1,"wal-append.queue_time_us":573}
I20240503 02:15:01.500376 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(12da2248f416404da29a4557b90daef7): 4838 bytes on disk
I20240503 02:15:01.501773 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(12da2248f416404da29a4557b90daef7) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.506058 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(73ae9bc7a96149259bd807369c2daacd): 4838 bytes on disk
I20240503 02:15:01.506709 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(73ae9bc7a96149259bd807369c2daacd) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4}
W20240503 02:15:01.541131 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:01.567737 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:01.643482 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:01.744254 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:01.756415 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(a2aac20a18f4444a9ff79642b23c89d9): perf score=0.033380
I20240503 02:15:01.806847 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(20366a3e8dcc4996bfdd1d60ef1be077): perf score=0.033403
W20240503 02:15:01.861454 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:01.904870 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(20366a3e8dcc4996bfdd1d60ef1be077) complete. Timing: real 0.097s user 0.048s sys 0.020s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":9825,"dirs.run_cpu_time_us":288,"dirs.run_wall_time_us":6030,"drs_written":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1621,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":13678,"threads_started":2,"wal-append.queue_time_us":6983}
I20240503 02:15:01.908037 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(20366a3e8dcc4996bfdd1d60ef1be077): 4923 bytes on disk
I20240503 02:15:01.912461 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(20366a3e8dcc4996bfdd1d60ef1be077) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":591,"lbm_reads_lt_1ms":4}
W20240503 02:15:01.922619 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
I20240503 02:15:01.955551 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(a2aac20a18f4444a9ff79642b23c89d9) complete. Timing: real 0.198s user 0.083s sys 0.049s Metrics: {"bytes_written":6144,"cfile_init":1,"compiler_manager_pool.queue_time_us":14534,"dirs.queue_time_us":16529,"dirs.run_cpu_time_us":340,"dirs.run_wall_time_us":1156,"drs_written":1,"lbm_read_time_us":151,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1068,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":38088,"threads_started":3,"wal-append.queue_time_us":6318}
I20240503 02:15:01.959568 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(a2aac20a18f4444a9ff79642b23c89d9): 4923 bytes on disk
I20240503 02:15:01.960718 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(a2aac20a18f4444a9ff79642b23c89d9) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":127,"lbm_reads_lt_1ms":4}
I20240503 02:15:01.965114 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(36457442a8e24ece9a1c2d8cdab89b78): perf score=0.033400
W20240503 02:15:02.021247 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:02.022536 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(36457442a8e24ece9a1c2d8cdab89b78) complete. Timing: real 0.057s user 0.039s sys 0.013s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":306,"dirs.run_wall_time_us":1905,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":974,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":308,"threads_started":1,"wal-append.queue_time_us":508}
I20240503 02:15:02.025350 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(36457442a8e24ece9a1c2d8cdab89b78): 4583 bytes on disk
I20240503 02:15:02.026031 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(36457442a8e24ece9a1c2d8cdab89b78) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
W20240503 02:15:02.029706 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:02.118250 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:02.130375 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:02.144651 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:02.159931 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:02.230973 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:02.563958 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:02.663246 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:02.694583 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:02.785135 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:02.828038 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:02.899322 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:02.969799 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:02.976481 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 176: this message will repeat every 5th retry.
W20240503 02:15:03.042126 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.092746 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.129037 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.140300 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.149575 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.254436 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.277761 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.277891 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 89 similar messages]
I20240503 02:15:03.290186 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(0dc2733b737b431684ce439c3d3fa762): perf score=0.033384
I20240503 02:15:03.330624 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(0dc2733b737b431684ce439c3d3fa762) complete. Timing: real 0.040s user 0.023s sys 0.012s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":492,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":2256,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":947,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":650,"threads_started":2,"wal-append.queue_time_us":455}
I20240503 02:15:03.333153 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(0dc2733b737b431684ce439c3d3fa762): 4581 bytes on disk
I20240503 02:15:03.333863 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(0dc2733b737b431684ce439c3d3fa762) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":90,"lbm_reads_lt_1ms":4}
W20240503 02:15:03.346100 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.359388 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:03.371767 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:03.383841 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:03.386025 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:03.416623 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
I20240503 02:15:03.527828 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(35aea508f59f4348b05d93a5afdde4ba): perf score=0.033375
I20240503 02:15:03.582240 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(35aea508f59f4348b05d93a5afdde4ba) complete. Timing: real 0.054s user 0.039s sys 0.013s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":432,"dirs.run_cpu_time_us":263,"dirs.run_wall_time_us":2508,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":863,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":550,"threads_started":2,"wal-append.queue_time_us":417}
I20240503 02:15:03.584347 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(35aea508f59f4348b05d93a5afdde4ba): 4836 bytes on disk
I20240503 02:15:03.585150 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(35aea508f59f4348b05d93a5afdde4ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
W20240503 02:15:03.651083 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:15:03.669828 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:03.698235 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:03.725225 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:15:03.726099 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.909435 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:03.919188 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:03.949653 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:03.954118 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.012764 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.109483 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.158515 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.171587 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.174269 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:04.184546 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.321056 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:04.322053 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.347296 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:04.440282 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:04.512607 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.558326 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:04.564563 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.638336 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.672674 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:04.739663 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:15:04.847685 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(da288f76df0a409eb01ee431d3b83b49): perf score=0.033354
I20240503 02:15:04.894425 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(da288f76df0a409eb01ee431d3b83b49) complete. Timing: real 0.046s user 0.026s sys 0.016s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":455,"dirs.run_cpu_time_us":218,"dirs.run_wall_time_us":2145,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":807,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":621,"threads_started":2,"wal-append.queue_time_us":414}
I20240503 02:15:04.896387 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(da288f76df0a409eb01ee431d3b83b49): 4923 bytes on disk
I20240503 02:15:04.897054 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(da288f76df0a409eb01ee431d3b83b49) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240503 02:15:04.985648 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:05.178140 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.182816 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:15:05.189875 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(e94a48cad1d648b18d709b46566d810b): perf score=0.033400
W20240503 02:15:05.190244 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:05.227811 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 181: this message will repeat every 5th retry.
W20240503 02:15:05.245930 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:15:05.254110 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(e94a48cad1d648b18d709b46566d810b) complete. Timing: real 0.064s user 0.051s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":414,"dirs.run_cpu_time_us":225,"dirs.run_wall_time_us":1968,"drs_written":1,"lbm_read_time_us":115,"lbm_reads_lt_1ms":4,"lbm_write_time_us":647,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":564,"threads_started":2,"wal-append.queue_time_us":412}
I20240503 02:15:05.256171 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(e94a48cad1d648b18d709b46566d810b): 4839 bytes on disk
I20240503 02:15:05.256794 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(e94a48cad1d648b18d709b46566d810b) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":81,"lbm_reads_lt_1ms":4}
W20240503 02:15:05.315673 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.342367 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.471119 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.567397 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.568188 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.597483 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:15:05.655194 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(5aa7ed62ebf44431a1a6db26ef73052a): perf score=0.033363
W20240503 02:15:05.671032 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
I20240503 02:15:05.710067 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(5aa7ed62ebf44431a1a6db26ef73052a) complete. Timing: real 0.055s user 0.029s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":376,"dirs.run_cpu_time_us":237,"dirs.run_wall_time_us":1861,"drs_written":1,"lbm_read_time_us":70,"lbm_reads_lt_1ms":4,"lbm_write_time_us":759,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":534,"threads_started":2,"wal-append.queue_time_us":411}
I20240503 02:15:05.712071 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(5aa7ed62ebf44431a1a6db26ef73052a): 4923 bytes on disk
I20240503 02:15:05.712797 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(5aa7ed62ebf44431a1a6db26ef73052a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":114,"lbm_reads_lt_1ms":4}
W20240503 02:15:05.754673 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.757905 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.788951 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.805440 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:05.851815 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.899626 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:05.943689 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:05.946182 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:06.023370 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:15:06.108935 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(390ad47a9a534abb81d59dc7604c72d3): perf score=0.033339
W20240503 02:15:06.109277 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:15:06.154739 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(390ad47a9a534abb81d59dc7604c72d3) complete. Timing: real 0.045s user 0.034s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1130,"dirs.run_cpu_time_us":317,"dirs.run_wall_time_us":1828,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1139,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1191,"threads_started":2,"wal-append.queue_time_us":442}
I20240503 02:15:06.156884 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(390ad47a9a534abb81d59dc7604c72d3): 4583 bytes on disk
I20240503 02:15:06.157544 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(390ad47a9a534abb81d59dc7604c72d3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4}
W20240503 02:15:06.214089 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:15:06.266383 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(49ce473013a942698081c389ace54b32): perf score=0.033353
I20240503 02:15:06.313973 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(49ce473013a942698081c389ace54b32) complete. Timing: real 0.047s user 0.026s sys 0.018s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":377,"dirs.run_cpu_time_us":232,"dirs.run_wall_time_us":1899,"drs_written":1,"lbm_read_time_us":65,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":548,"threads_started":2,"wal-append.queue_time_us":476}
I20240503 02:15:06.317013 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(49ce473013a942698081c389ace54b32): 4921 bytes on disk
I20240503 02:15:06.317739 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(49ce473013a942698081c389ace54b32) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":85,"lbm_reads_lt_1ms":4}
W20240503 02:15:06.418151 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:06.420552 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:06.482813 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:06.529886 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:06.549666 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:06.549888 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:06.571650 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:06.629843 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:06.723631 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:06.739387 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:06.787001 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:07.019874 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:07.032176 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:07.045542 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:07.091579 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:07.226696 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:07.276381 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:07.289075 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:07.381036 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:07.426769 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:15:07.580214 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(af7beeafa08f477796e5070fbc177dcc): perf score=0.033363
I20240503 02:15:07.631680 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(af7beeafa08f477796e5070fbc177dcc) complete. Timing: real 0.051s user 0.031s sys 0.012s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":488,"dirs.run_cpu_time_us":283,"dirs.run_wall_time_us":1763,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":901,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":663,"threads_started":2,"wal-append.queue_time_us":492}
I20240503 02:15:07.634308 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(af7beeafa08f477796e5070fbc177dcc): 4923 bytes on disk
I20240503 02:15:07.635035 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(af7beeafa08f477796e5070fbc177dcc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
I20240503 02:15:07.650898 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(476f6e414dae4db3ba9a187adad75f8c): perf score=0.033348
I20240503 02:15:07.671496 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(5c762709f66f44b98089e69cbace1570): perf score=0.033354
W20240503 02:15:07.672585 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 186: this message will repeat every 5th retry.
W20240503 02:15:07.684252 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:07.694177 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
I20240503 02:15:07.698300 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(476f6e414dae4db3ba9a187adad75f8c) complete. Timing: real 0.047s user 0.035s sys 0.007s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":1071,"dirs.run_cpu_time_us":227,"dirs.run_wall_time_us":2980,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":788,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":1231,"threads_started":2,"wal-append.queue_time_us":729}
I20240503 02:15:07.700794 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(476f6e414dae4db3ba9a187adad75f8c): 4921 bytes on disk
I20240503 02:15:07.701752 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(476f6e414dae4db3ba9a187adad75f8c) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":145,"lbm_reads_lt_1ms":4}
I20240503 02:15:07.761902 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(5c762709f66f44b98089e69cbace1570) complete. Timing: real 0.090s user 0.058s sys 0.017s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":1375,"dirs.run_cpu_time_us":231,"dirs.run_wall_time_us":1601,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":911,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1734,"threads_started":2,"wal-append.queue_time_us":703}
I20240503 02:15:07.766852 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(5c762709f66f44b98089e69cbace1570): 4583 bytes on disk
I20240503 02:15:07.768883 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(5c762709f66f44b98089e69cbace1570) complete. Timing: real 0.002s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":171,"lbm_reads_lt_1ms":4}
W20240503 02:15:07.784624 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:07.882112 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:07.891287 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:07.935287 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.010490 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.027206 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.055140 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.122097 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.181492 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.200764 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:08.242851 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.285091 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.357801 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:15:08.417996 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:08.560145 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:08.565927 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.567472 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:08.672627 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:08.683547 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:08.684058 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:15:08.697229 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:08.758841 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:08.780474 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:08.781251 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.804705 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:08.946586 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:08.948977 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:09.056667 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:09.063294 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:09.140003 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:09.194177 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:09.223655 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:09.255156 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:09.348378 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:09.349529 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:09.463783 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:09.483819 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:09.505507 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:09.517738 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
I20240503 02:15:09.534981 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(31f72bc850c74289aa3a59ecc3dcc287): perf score=0.033354
I20240503 02:15:09.572659 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(31f72bc850c74289aa3a59ecc3dcc287) complete. Timing: real 0.037s user 0.026s sys 0.011s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":365,"dirs.run_cpu_time_us":193,"dirs.run_wall_time_us":1737,"drs_written":1,"lbm_read_time_us":56,"lbm_reads_lt_1ms":4,"lbm_write_time_us":761,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":562,"threads_started":2,"wal-append.queue_time_us":456}
I20240503 02:15:09.574765 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(31f72bc850c74289aa3a59ecc3dcc287): 4838 bytes on disk
I20240503 02:15:09.575358 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(31f72bc850c74289aa3a59ecc3dcc287) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4}
W20240503 02:15:09.576823 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:09.751617 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:09.755132 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:09.776731 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:09.794008 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:09.966177 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.008119 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 191: this message will repeat every 5th retry.
W20240503 02:15:10.033869 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.336776 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:10.370980 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.444756 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.448920 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:15:10.449713 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:10.502491 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.589252 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.597136 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.684737 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.732678 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:10.851943 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.857582 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.878026 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:10.920715 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:10.963168 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:11.092068 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:11.221904 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.247061 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.296456 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:11.298394 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.415994 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:11.496016 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.573864 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.593770 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.664261 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:11.749373 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.756361 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:11.796648 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.806413 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:11.873045 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.882920 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.905263 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.958760 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:11.976873 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:11.994340 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:11.994889 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:12.066020 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:12.277770 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:12.295876 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:12.516872 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:12.661692 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:12.674963 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 196: this message will repeat every 5th retry.
W20240503 02:15:12.800848 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:12.866598 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:12.905104 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:12.974941 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.099895 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.139608 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:13.186069 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.199398 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.290504 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.301916 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:15:13.356704 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(674ea54f2cbd4c438cab5a7723c5a2c6): perf score=0.033343
W20240503 02:15:13.384390 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:15:13.408905 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(674ea54f2cbd4c438cab5a7723c5a2c6) complete. Timing: real 0.052s user 0.041s sys 0.005s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":444,"dirs.run_cpu_time_us":310,"dirs.run_wall_time_us":2243,"drs_written":1,"lbm_read_time_us":72,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5881,"threads_started":2,"wal-append.queue_time_us":5725}
I20240503 02:15:13.411936 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(674ea54f2cbd4c438cab5a7723c5a2c6): 4923 bytes on disk
W20240503 02:15:13.412657 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
I20240503 02:15:13.414844 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(674ea54f2cbd4c438cab5a7723c5a2c6) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
W20240503 02:15:13.434798 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:13.458039 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:13.460150 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.482897 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.530750 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:13.606846 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.614878 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:15:13.687619 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(13e83ce8820b491992c041a8c21fa3df): perf score=0.033358
I20240503 02:15:13.727378 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(13e83ce8820b491992c041a8c21fa3df) complete. Timing: real 0.039s user 0.028s sys 0.008s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":928,"dirs.run_cpu_time_us":342,"dirs.run_wall_time_us":2504,"drs_written":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4,"lbm_write_time_us":690,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":955,"threads_started":2,"wal-append.queue_time_us":412}
I20240503 02:15:13.729552 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(13e83ce8820b491992c041a8c21fa3df): 4838 bytes on disk
I20240503 02:15:13.730254 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(13e83ce8820b491992c041a8c21fa3df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
W20240503 02:15:13.737252 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:15:13.781209 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:13.804291 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:13.878867 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 02:15:13.921269 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:13.985200 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
I20240503 02:15:14.036826 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(ee635c5c4d2c4befa0283cb8226f28d7): perf score=0.033395
W20240503 02:15:14.046530 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:14.082801 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.099771 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
I20240503 02:15:14.104691 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(ee635c5c4d2c4befa0283cb8226f28d7) complete. Timing: real 0.068s user 0.048s sys 0.019s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":421,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":1896,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":765,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":546,"threads_started":2,"wal-append.queue_time_us":417}
I20240503 02:15:14.107216 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(ee635c5c4d2c4befa0283cb8226f28d7): 4923 bytes on disk
I20240503 02:15:14.108126 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(ee635c5c4d2c4befa0283cb8226f28d7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":107,"lbm_reads_lt_1ms":4}
I20240503 02:15:14.110236 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(2fadc7e9c7604b5d8f7b8751b6ca78a7): perf score=0.033396
W20240503 02:15:14.116940 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:15:14.149670 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(2fadc7e9c7604b5d8f7b8751b6ca78a7) complete. Timing: real 0.039s user 0.027s sys 0.010s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":64,"dirs.run_cpu_time_us":287,"dirs.run_wall_time_us":1685,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":888,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":316,"threads_started":1,"wal-append.queue_time_us":456}
I20240503 02:15:14.151997 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(2fadc7e9c7604b5d8f7b8751b6ca78a7): 4836 bytes on disk
I20240503 02:15:14.152680 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(2fadc7e9c7604b5d8f7b8751b6ca78a7) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":93,"lbm_reads_lt_1ms":4}
W20240503 02:15:14.186159 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.200042 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.223943 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.244505 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:14.355379 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.358832 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:15:14.359804 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:14.360832 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.398744 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.454059 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.486713 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:14.509558 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:14.538421 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.774976 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:14.792479 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:14.945902 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.083501 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.209110 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.211706 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:15.224685 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 201: this message will repeat every 5th retry.
W20240503 02:15:15.341060 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.424934 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.436673 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.441177 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.456787 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:15.464675 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:15:15.759141 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:15.762477 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.785379 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.798961 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:15.827529 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:15.899060 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.961715 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:15.999979 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:16.035835 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:16.118256 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:16.147240 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:16.170084 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:15:16.170617 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(ceb177d6952e4c978c8be93b5b026952): perf score=0.033340
I20240503 02:15:16.211283 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(ceb177d6952e4c978c8be93b5b026952) complete. Timing: real 0.040s user 0.034s sys 0.003s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":375,"dirs.run_cpu_time_us":214,"dirs.run_wall_time_us":1693,"drs_written":1,"lbm_read_time_us":60,"lbm_reads_lt_1ms":4,"lbm_write_time_us":822,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":492,"threads_started":2,"wal-append.queue_time_us":345}
I20240503 02:15:16.213614 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(ceb177d6952e4c978c8be93b5b026952): 4583 bytes on disk
I20240503 02:15:16.214428 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(ceb177d6952e4c978c8be93b5b026952) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
W20240503 02:15:16.395697 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:16.458978 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:16.569851 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:16.579699 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:16.586432 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:16.673828 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:16.692799 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(d2fb90eef6224ed1a1dcc57c71683bc3): perf score=0.033359
W20240503 02:15:16.710137 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:16.725175 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:16.728693 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:16.763742 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(d2fb90eef6224ed1a1dcc57c71683bc3) complete. Timing: real 0.071s user 0.040s sys 0.029s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":416,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1999,"drs_written":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4,"lbm_write_time_us":693,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":1453,"threads_started":2,"wal-append.queue_time_us":1234}
I20240503 02:15:16.766320 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(d2fb90eef6224ed1a1dcc57c71683bc3): 4838 bytes on disk
I20240503 02:15:16.767266 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(d2fb90eef6224ed1a1dcc57c71683bc3) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20240503 02:15:16.779479 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(be61de73e08c4998a51f02cafa46d421): perf score=0.033356
W20240503 02:15:16.790330 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:16.815449 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
I20240503 02:15:16.835505 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(be61de73e08c4998a51f02cafa46d421) complete. Timing: real 0.052s user 0.040s sys 0.004s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":2952,"dirs.run_cpu_time_us":215,"dirs.run_wall_time_us":2539,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":913,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":3453,"threads_started":2,"wal-append.queue_time_us":377}
I20240503 02:15:16.837735 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(be61de73e08c4998a51f02cafa46d421): 4583 bytes on disk
I20240503 02:15:16.838325 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(be61de73e08c4998a51f02cafa46d421) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":71,"lbm_reads_lt_1ms":4}
W20240503 02:15:16.955421 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.025327 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:17.052115 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.081311 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.089591 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.128787 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.157430 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:17.209877 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:17.451330 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.577436 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:17.646831 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.683940 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 206: this message will repeat every 5th retry.
W20240503 02:15:17.772420 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.819233 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.841786 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.895561 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:17.953263 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:18.053822 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:18.176390 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:18.200321 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:18.266822 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:18.297361 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:18.328670 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:18.346091 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:18.413269 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:18.439584 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:15:18.440666 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:18.472527 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:18.505721 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:18.641286 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:18.756994 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:18.793449 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:18.853946 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:15:18.923883 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:18.952111 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:18.953631 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
I20240503 02:15:19.037149 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling FlushMRSOp(0a5125fbc8ec4ba887abce94a84efbdc): perf score=0.033373
I20240503 02:15:19.077443 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: FlushMRSOp(0a5125fbc8ec4ba887abce94a84efbdc) complete. Timing: real 0.040s user 0.024s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":395,"dirs.run_cpu_time_us":255,"dirs.run_wall_time_us":1845,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":697,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":571,"threads_started":2,"wal-append.queue_time_us":421}
I20240503 02:15:19.079841 13356 maintenance_manager.cc:419] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Scheduling UndoDeltaBlockGCOp(0a5125fbc8ec4ba887abce94a84efbdc): 4923 bytes on disk
I20240503 02:15:19.080435 13247 maintenance_manager.cc:643] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: UndoDeltaBlockGCOp(0a5125fbc8ec4ba887abce94a84efbdc) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4}
W20240503 02:15:19.158722 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:19.175140 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:19.211741 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:19.227258 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:19.229892 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:19.237145 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:19.269510 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:19.349438 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:15:19.359375 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(9f8b006a679a4b8d8d3551310b78fb3a): perf score=0.033397
W20240503 02:15:19.445588 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
I20240503 02:15:19.446980 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(9f8b006a679a4b8d8d3551310b78fb3a) complete. Timing: real 0.087s user 0.054s sys 0.015s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":14723,"dirs.run_cpu_time_us":310,"dirs.run_wall_time_us":18033,"drs_written":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4,"lbm_write_time_us":978,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":24320,"threads_started":2,"wal-append.queue_time_us":9840}
I20240503 02:15:19.449518 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(9f8b006a679a4b8d8d3551310b78fb3a): 4581 bytes on disk
I20240503 02:15:19.450234 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(9f8b006a679a4b8d8d3551310b78fb3a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":96,"lbm_reads_lt_1ms":4}
W20240503 02:15:19.520473 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:19.544399 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:19.551474 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:19.651453 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:19.687816 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:19.714133 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:19.773188 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:19.787673 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:19.832990 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:19.842902 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:19.877290 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.097496 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.146011 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.246157 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:20.280527 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 211: this message will repeat every 5th retry.
W20240503 02:15:20.342622 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.357936 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.407470 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.520823 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.544661 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:20.570595 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:20.671933 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:15:20.722637 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.816251 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:20.860368 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.887372 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:20.916244 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.956341 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:20.982486 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:21.196771 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:21.283569 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:21.289275 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:21.368371 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:21.567785 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:21.603156 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:21.708967 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:21.714735 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:21.780668 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:21.790604 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:21.846235 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:21.896239 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:21.909622 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:21.925112 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:21.944003 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.023691 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.039719 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.147693 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.166100 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.203012 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:22.215236 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.383273 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:22.386060 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.389487 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.569082 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.618099 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:22.742542 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:22.745512 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 216: this message will repeat every 5th retry.
W20240503 02:15:22.776432 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:22.870308 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:23.081410 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:23.123471 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:23.175895 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:23.213361 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:23.240211 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:23.277705 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:23.329893 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:23.336664 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:23.440738 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:23.461642 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:15:23.462814 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:23.523070 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:15:23.542521 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:23.745141 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:23.751058 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:23.822322 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:23.938448 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:15:24.011973 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.025075 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 02:15:24.026640 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:24.037089 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:24.159148 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:24.231055 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.259742 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:24.259719 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:24.352275 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:24.379038 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.380620 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.408000 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.450331 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:24.453261 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.535864 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:15:24.570295 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(9f8b006a679a4b8d8d3551310b78fb3a): perf score=0.033361
I20240503 02:15:24.607589 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(9f8b006a679a4b8d8d3551310b78fb3a) complete. Timing: real 0.037s user 0.017s sys 0.016s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":397,"dirs.run_cpu_time_us":274,"dirs.run_wall_time_us":1596,"drs_written":1,"lbm_read_time_us":51,"lbm_reads_lt_1ms":4,"lbm_write_time_us":714,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":927,"threads_started":2,"wal-append.queue_time_us":819}
I20240503 02:15:24.610936 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(9f8b006a679a4b8d8d3551310b78fb3a): 4581 bytes on disk
I20240503 02:15:24.611773 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(9f8b006a679a4b8d8d3551310b78fb3a) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":125,"lbm_reads_lt_1ms":4}
W20240503 02:15:24.648226 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.668320 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.730177 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:24.737228 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:24.749756 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.799780 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:24.951521 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
I20240503 02:15:24.994458 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(13e83ce8820b491992c041a8c21fa3df): perf score=0.033361
W20240503 02:15:25.016700 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:25.047011 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:15:25.056318 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(13e83ce8820b491992c041a8c21fa3df) complete. Timing: real 0.062s user 0.038s sys 0.018s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":423,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1499,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":795,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":533,"threads_started":2,"wal-append.queue_time_us":367}
I20240503 02:15:25.059182 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(13e83ce8820b491992c041a8c21fa3df): 4838 bytes on disk
I20240503 02:15:25.059913 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(13e83ce8820b491992c041a8c21fa3df) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4}
W20240503 02:15:25.076303 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.143343 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.198836 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 221: this message will repeat every 5th retry.
W20240503 02:15:25.268625 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:25.406859 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.439662 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:25.496516 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.583159 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.640679 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.703482 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:25.721180 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.724934 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:25.802328 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.906236 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:25.986265 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:26.097976 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:26.103199 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:26.103937 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:26.117563 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:26.118660 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:26.380831 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:26.389919 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:26.404556 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:26.497442 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:26.593829 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:26.685447 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:26.750257 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:26.840526 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:26.886842 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:26.920219 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.002111 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:27.046802 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.067668 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.094589 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:27.137082 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.158988 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:27.210601 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.239457 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.292392 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
I20240503 02:15:27.330929 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(af7beeafa08f477796e5070fbc177dcc): perf score=0.033334
I20240503 02:15:27.385632 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(af7beeafa08f477796e5070fbc177dcc) complete. Timing: real 0.054s user 0.032s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":406,"dirs.run_cpu_time_us":280,"dirs.run_wall_time_us":1917,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":882,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":5453,"threads_started":2,"wal-append.queue_time_us":5328}
I20240503 02:15:27.388136 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(af7beeafa08f477796e5070fbc177dcc): 4923 bytes on disk
I20240503 02:15:27.397239 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(af7beeafa08f477796e5070fbc177dcc) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":129,"lbm_reads_lt_1ms":4}
W20240503 02:15:27.415867 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:27.434446 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.480624 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:15:27.556432 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(fc0a049eeb0047558d3c9eb370467c53): perf score=0.033344
I20240503 02:15:27.593884 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(fc0a049eeb0047558d3c9eb370467c53) complete. Timing: real 0.037s user 0.027s sys 0.008s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":511,"dirs.run_cpu_time_us":267,"dirs.run_wall_time_us":1806,"drs_written":1,"lbm_read_time_us":53,"lbm_reads_lt_1ms":4,"lbm_write_time_us":877,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":624,"threads_started":2,"wal-append.queue_time_us":447}
I20240503 02:15:27.596351 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(fc0a049eeb0047558d3c9eb370467c53): 4583 bytes on disk
W20240503 02:15:27.596382 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
I20240503 02:15:27.596985 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(fc0a049eeb0047558d3c9eb370467c53) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4}
W20240503 02:15:27.692870 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:27.727239 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 226: this message will repeat every 5th retry.
W20240503 02:15:27.774196 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.871628 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.914640 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:27.967343 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.003686 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:28.144377 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.193876 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.243480 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:28.246168 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.251822 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.286484 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:28.485951 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.489037 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 02:15:28.490193 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:28.534761 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.577198 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:15:28.578171 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:28.594460 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:28.678982 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:28.773123 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:28.829527 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.901209 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:28.904774 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:28.938513 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:15:29.088361 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:29.201752 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:15:29.243981 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:29.306156 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:29.346742 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:29.370728 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:29.380509 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:15:29.406595 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(5aa7ed62ebf44431a1a6db26ef73052a): perf score=0.033378
W20240503 02:15:29.425933 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:29.449448 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:15:29.453706 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(5aa7ed62ebf44431a1a6db26ef73052a) complete. Timing: real 0.047s user 0.028s sys 0.015s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":433,"dirs.run_cpu_time_us":324,"dirs.run_wall_time_us":1795,"drs_written":1,"lbm_read_time_us":50,"lbm_reads_lt_1ms":4,"lbm_write_time_us":815,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":665,"threads_started":2,"wal-append.queue_time_us":487}
I20240503 02:15:29.456281 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(5aa7ed62ebf44431a1a6db26ef73052a): 4923 bytes on disk
I20240503 02:15:29.457150 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(5aa7ed62ebf44431a1a6db26ef73052a) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":117,"lbm_reads_lt_1ms":4}
W20240503 02:15:29.461599 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:29.478847 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:29.487339 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:29.523102 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:29.551904 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:29.554792 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:29.749256 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:15:29.807629 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:29.942253 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:29.963770 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:30.144552 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.155368 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.156423 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 231: this message will repeat every 5th retry.
W20240503 02:15:30.209208 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.278215 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.312269 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:30.400820 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.441030 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.477810 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:30.490932 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.521283 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.603646 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:15:30.622313 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling FlushMRSOp(49ce473013a942698081c389ace54b32): perf score=0.033367
I20240503 02:15:30.686362 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: FlushMRSOp(49ce473013a942698081c389ace54b32) complete. Timing: real 0.064s user 0.037s sys 0.024s Metrics: {"bytes_written":6144,"cfile_init":1,"dirs.queue_time_us":406,"dirs.run_cpu_time_us":249,"dirs.run_wall_time_us":2091,"drs_written":1,"lbm_read_time_us":78,"lbm_reads_lt_1ms":4,"lbm_write_time_us":849,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":689,"thread_start_us":593,"threads_started":2,"wal-append.queue_time_us":444}
I20240503 02:15:30.689111 13224 maintenance_manager.cc:419] P f93b103af083471b8e8c0376e9148aad: Scheduling UndoDeltaBlockGCOp(49ce473013a942698081c389ace54b32): 4921 bytes on disk
I20240503 02:15:30.691833 13111 maintenance_manager.cc:643] P f93b103af083471b8e8c0376e9148aad: UndoDeltaBlockGCOp(49ce473013a942698081c389ace54b32) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":291,"lbm_reads_lt_1ms":4}
W20240503 02:15:30.702803 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.757371 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:30.816052 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:30.909883 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:31.086831 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:15:31.182121 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling FlushMRSOp(ceb177d6952e4c978c8be93b5b026952): perf score=0.033385
W20240503 02:15:31.185930 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:31.211136 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:15:31.223773 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(5547b33324db46c886287b4ee5035e43): perf score=0.033346
W20240503 02:15:31.238430 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:15:31.261945 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: FlushMRSOp(ceb177d6952e4c978c8be93b5b026952) complete. Timing: real 0.080s user 0.052s sys 0.024s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":671,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":944,"drs_written":1,"lbm_read_time_us":82,"lbm_reads_lt_1ms":4,"lbm_write_time_us":874,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":5172,"threads_started":2,"wal-append.queue_time_us":4810}
I20240503 02:15:31.266712 13087 maintenance_manager.cc:419] P 8d454a3607d042dc9559ed59c3ccd84b: Scheduling UndoDeltaBlockGCOp(ceb177d6952e4c978c8be93b5b026952): 4583 bytes on disk
I20240503 02:15:31.268044 12977 maintenance_manager.cc:643] P 8d454a3607d042dc9559ed59c3ccd84b: UndoDeltaBlockGCOp(ceb177d6952e4c978c8be93b5b026952) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":152,"lbm_reads_lt_1ms":4}
W20240503 02:15:31.276378 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:31.277695 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:15:31.277666 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(5547b33324db46c886287b4ee5035e43) complete. Timing: real 0.054s user 0.030s sys 0.018s Metrics: {"bytes_written":6062,"cfile_init":1,"dirs.queue_time_us":421,"dirs.run_cpu_time_us":187,"dirs.run_wall_time_us":2917,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1121,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":635,"thread_start_us":1418,"threads_started":2,"wal-append.queue_time_us":1264}
I20240503 02:15:31.279814 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(5547b33324db46c886287b4ee5035e43): 4583 bytes on disk
I20240503 02:15:31.280575 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(5547b33324db46c886287b4ee5035e43) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":88,"lbm_reads_lt_1ms":4}
W20240503 02:15:31.333774 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:31.448246 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:31.494153 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:31.528878 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:31.594409 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:31.713822 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:31.782546 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
I20240503 02:15:31.786664 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling FlushMRSOp(35aea508f59f4348b05d93a5afdde4ba): perf score=0.033361
W20240503 02:15:31.819831 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:31.823815 12973 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.68:35025, user_credentials={real_user=slave}} blocked reactor thread for 48643us
W20240503 02:15:31.878491 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
I20240503 02:15:31.884904 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: FlushMRSOp(35aea508f59f4348b05d93a5afdde4ba) complete. Timing: real 0.098s user 0.049s sys 0.024s Metrics: {"bytes_written":6152,"cfile_init":1,"dirs.queue_time_us":737,"dirs.run_cpu_time_us":267,"dirs.run_wall_time_us":2477,"drs_written":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4,"lbm_write_time_us":953,"lbm_writes_lt_1ms":23,"peak_mem_usage":0,"rows_written":676,"thread_start_us":891,"threads_started":2,"wal-append.queue_time_us":427}
I20240503 02:15:31.887657 12954 maintenance_manager.cc:419] P e3cbe34992dc4b77b9aa9e9a4c143e78: Scheduling UndoDeltaBlockGCOp(35aea508f59f4348b05d93a5afdde4ba): 4836 bytes on disk
I20240503 02:15:31.888365 12844 maintenance_manager.cc:643] P e3cbe34992dc4b77b9aa9e9a4c143e78: UndoDeltaBlockGCOp(35aea508f59f4348b05d93a5afdde4ba) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":106,"lbm_reads_lt_1ms":4}
W20240503 02:15:31.898664 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:31.926995 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:31.952773 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:31.954408 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:32.013135 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:32.030362 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:32.035634 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:32.198971 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:32.209800 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:32.379954 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:32.566181 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 236: this message will repeat every 5th retry.
W20240503 02:15:32.608902 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:32.710959 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:32.774510 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:32.811568 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:32.823525 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:32.825210 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:32.899565 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:33.063879 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.103351 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.213227 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:33.254820 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.268793 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.351075 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.444890 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.494379 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.496261 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 02:15:33.536924 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:33.542738 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.585521 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:33.586599 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:33.664433 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:33.729328 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:33.748004 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:33.905265 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:33.940085 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.940138 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:33.949194 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:33.959278 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:15:34.026679 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:34.043493 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:34.082055 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:34.166143 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:34.178153 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:34.223937 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:34.254572 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:34.272727 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:34.314612 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:34.347332 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:34.386451 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:34.390767 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:34.393103 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:34.543151 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:34.612253 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:34.662279 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:34.837013 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:34.842042 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:15:35.146312 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:35.170123 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.221168 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 241: this message will repeat every 5th retry.
W20240503 02:15:35.277987 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.281374 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.357836 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.363588 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.460727 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.467720 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.524786 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:35.576248 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.702891 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:35.872829 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:35.933053 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:35.995383 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:36.001096 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.010613 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:36.024168 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:36.108419 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:36.198004 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.212834 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.245441 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:36.262127 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.309880 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:36.458575 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:36.483517 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.500859 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.561199 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.586465 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.607003 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.674324 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:15:36.714520 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.737156 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:36.748804 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:36.949676 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:36.968279 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:37.002501 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:37.007611 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:37.071031 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:37.099848 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:37.111708 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:37.205693 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:37.546499 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 246: this message will repeat every 5th retry.
W20240503 02:15:37.595388 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:37.605746 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:37.683800 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:37.697077 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:37.782565 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:37.792304 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:37.848294 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:37.865605 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.061650 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.084808 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:38.120174 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:38.345350 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.412123 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.449503 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.539286 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.545243 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:15:38.546461 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:38.576256 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:15:38.584580 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:38.592562 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:38.593699 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:38.670892 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:38.702502 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:38.750571 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:38.777695 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.786551 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.789631 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:38.936519 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:38.990093 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:15:38.991498 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:38.993383 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:39.042462 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:39.096689 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:39.209287 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:39.236284 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:39.237483 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:39.246042 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:39.271081 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:39.271265 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:39.348901 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:39.384060 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:39.478413 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:39.487550 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:39.516661 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:39.572597 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:39.742239 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:39.835929 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:39.901427 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:39.991866 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.003449 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:15:40.004400 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:40.085459 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.133273 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 251: this message will repeat every 5th retry.
W20240503 02:15:40.337492 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.391911 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.409981 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.446121 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:40.512544 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.512812 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.524770 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.603696 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:40.809998 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:40.820024 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:40.874915 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:41.000011 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:41.000991 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.054783 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.082803 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:41.240224 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:41.285367 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.309234 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.376600 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:41.462729 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:41.483908 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:41.499347 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.578796 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.660905 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:41.669543 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.681349 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.685501 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.728291 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.750655 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:41.768013 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:41.804816 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:41.840880 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.890004 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:41.911607 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:42.313652 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:42.338570 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:42.346660 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:42.353014 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:42.418120 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:42.465222 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:42.551501 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:42.784793 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:42.815008 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 256: this message will repeat every 5th retry.
W20240503 02:15:42.876037 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:42.924253 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:42.978907 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:43.011196 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:43.076524 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:43.122471 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:43.205719 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:43.210011 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:43.259918 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:43.352702 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:43.352985 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:43.359658 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:43.420296 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:43.529937 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:43.544507 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:43.632020 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 02:15:43.661222 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:43.692643 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:43.730175 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.022197 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.097846 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.105160 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.119585 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:15:44.125042 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:44.183769 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.260545 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.284180 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:44.285485 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.285822 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.296039 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.352340 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.361807 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.365832 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.381258 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.387799 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:44.389324 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.506740 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:44.638724 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.738613 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.744431 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.772766 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:44.919576 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:44.947538 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:45.033794 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.040092 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.223417 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:45.226269 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 261: this message will repeat every 5th retry.
W20240503 02:15:45.235266 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.322002 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.462509 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:45.480423 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.486336 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.498816 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.580979 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:45.593983 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.741058 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.790966 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:45.825995 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:45.829931 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:45.900931 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:46.011849 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:46.068589 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:46.107847 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:46.286320 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:46.349283 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:46.438288 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:46.517561 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:46.553820 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:46.573479 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:46.624109 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:46.671136 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:46.679661 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:46.729915 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:46.814539 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:46.872166 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:46.893939 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:46.899714 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.019096 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.023864 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:15:47.143363 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.210649 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.308014 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:47.326213 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:47.337805 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.371686 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:47.524315 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:47.581601 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.612381 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.691087 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.730262 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 266: this message will repeat every 5th retry.
W20240503 02:15:47.844062 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.873214 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.919526 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:47.919487 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:47.937741 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:48.078913 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:48.128755 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:48.139246 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:48.197566 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:48.271850 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:48.387668 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:48.455740 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:48.481207 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:48.637636 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:15:48.641772 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:48.655826 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:48.712772 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:48.759493 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:48.778832 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:48.943629 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:48.961256 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:49.104360 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.167627 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:49.182788 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:15:49.230515 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:49.256078 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.293438 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:15:49.294572 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:49.351226 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:49.372124 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.407347 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.440155 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.446672 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.496182 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.588256 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:49.588536 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:15:49.653187 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:49.709666 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.822036 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.857858 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:49.876897 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.878684 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:49.879678 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:50.087141 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.090175 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 271: this message will repeat every 5th retry.
W20240503 02:15:50.095605 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.333519 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.367849 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.392939 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:50.394119 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:50.438488 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.573274 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.617395 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:50.624195 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.786687 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.828405 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:50.844529 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:50.911904 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:50.931631 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:50.956516 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:50.978964 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:51.064572 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:51.198746 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:51.207298 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:51.418988 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:51.544368 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:51.594383 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:51.595528 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:51.727522 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:51.733381 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:51.879801 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:51.891984 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:51.982977 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.024552 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:52.024710 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.076403 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.133478 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.176568 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:52.272154 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.293303 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:52.296249 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.318429 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.318621 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:52.352530 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.512291 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:52.513444 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.570443 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.609172 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:52.643453 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 276: this message will repeat every 5th retry.
W20240503 02:15:52.909464 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:52.909636 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.048000 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:53.075984 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.177508 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.229920 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:53.232372 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:53.252395 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:53.290194 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.299165 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:53.420488 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.511168 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.605538 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:53.613071 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.662953 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 02:15:53.672729 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:53.740094 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:53.852150 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:15:53.886051 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.892300 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:53.999305 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:54.039443 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:54.185802 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:54.244194 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:15:54.280934 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:54.354513 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:15:54.402339 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.490247 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.496894 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:54.500039 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:54.552914 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.572736 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.642851 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.730908 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.747236 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:54.750840 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.759550 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.797384 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.802197 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:54.854192 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.876315 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.895584 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:54.994043 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:55.168769 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:55.297430 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:55.374387 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:55.379652 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:55.387012 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:55.450664 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 281: this message will repeat every 5th retry.
W20240503 02:15:55.480501 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:15:55.512732 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:55.517050 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:55.555946 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:55.566843 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:55.613663 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:55.624425 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:55.780891 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:55.935602 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.044202 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:56.053215 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:56.187363 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:56.228895 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.375697 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.379832 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.383109 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.405292 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.420068 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.549566 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.583395 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:56.603379 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:56.855080 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:56.901455 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:56.932425 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:56.989183 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.059389 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.083971 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.125773 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:57.264683 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:15:57.269179 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.289919 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:57.297621 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.303886 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.398396 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.402140 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.425483 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.558574 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:57.604040 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:57.821502 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:57.851248 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:57.856467 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:57.947966 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.019567 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:58.037892 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.047922 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.062484 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 286: this message will repeat every 5th retry.
W20240503 02:15:58.134348 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.149866 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:58.239446 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.410697 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.528241 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.551862 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:58.562258 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:58.629997 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:58.672195 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.688589 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:15:58.734956 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:58.783931 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.795245 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.830890 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.842243 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:58.859496 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:58.967391 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:59.030805 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:59.185087 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:59.231400 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:59.266146 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.267454 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:15:59.332192 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.401630 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:15:59.457605 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.491336 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:59.549623 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.664330 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.781817 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:15:59.793470 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.816037 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.871582 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:15:59.879000 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:59.935400 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.950722 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.954288 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:15:59.970341 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:15:59.984707 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:00.188629 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:00.331418 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:00.342254 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:00.400938 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:00.428067 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:00.477603 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:00.503701 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:00.589761 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:00.625433 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:16:00.650741 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:00.686653 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:00.849938 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 291: this message will repeat every 5th retry.
W20240503 02:16:00.911640 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:00.916617 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:00.997357 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.136469 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:01.152521 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.253403 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.272383 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.329862 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:01.331143 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.348224 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.440367 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:01.499722 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:01.606974 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.716822 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.739408 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:01.952095 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:01.968044 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:01.970772 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.016974 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:02.094864 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.098845 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.255273 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.286890 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:02.326599 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.346212 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.437561 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:16:02.453572 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:02.454645 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.501363 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:02.507563 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.600664 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.776842 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:02.798296 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.809408 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.811661 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.818629 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:02.844380 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:02.868786 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:03.208218 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:03.235772 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:03.293726 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:03.312405 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:03.314175 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:03.366082 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:03.466053 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 296: this message will repeat every 5th retry.
W20240503 02:16:03.518333 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:03.640136 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:03.722486 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:03.729319 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 02:16:03.754489 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:03.778795 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:03.803701 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:03.830965 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:03.911193 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:03.914563 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:03.975666 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:16:04.086958 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:04.222086 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:04.283763 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:16:04.284893 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.348976 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:04.367300 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.370234 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.459282 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:04.484735 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:04.487115 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.712644 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.713735 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:04.767570 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.812130 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.850323 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.878132 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:04.883666 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:04.899438 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:16:05.015626 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:05.144959 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:05.206548 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:05.214253 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:05.255872 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:05.312944 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:05.559513 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:05.582659 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:05.589373 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:05.660676 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:05.733417 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:05.765470 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:05.811908 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:05.866142 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:05.897912 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:05.941699 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:05.947309 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:06.031576 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:06.222019 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:06.248349 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:06.311304 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:06.316490 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:06.332196 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 301: this message will repeat every 5th retry.
W20240503 02:16:06.380012 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:06.428081 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:06.428614 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:06.436934 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:06.472627 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:06.796481 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:06.924656 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:06.940981 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:06.967929 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:07.034365 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:07.039538 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:07.060093 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:07.090862 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:07.125015 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:07.260826 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:07.329700 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:07.391927 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:07.416585 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:07.464771 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:07.529114 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:07.534018 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:07.554580 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:07.594744 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:07.610078 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:07.705262 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:07.780882 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.024400 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.028100 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.168774 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:08.172665 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.226109 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.273036 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.411598 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:08.425933 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:08.556645 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:08.601372 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:08.610721 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.655364 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.658445 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.669304 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.813870 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:16:08.814888 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:08.817544 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:08.915020 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:08.918526 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:09.016177 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:09.018043 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:16:09.018978 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.148720 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:09.179006 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:09.183555 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 306: this message will repeat every 5th retry.
W20240503 02:16:09.379299 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:16:09.381227 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.529934 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:09.605795 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.627209 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:09.631199 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.698052 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.727685 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.755602 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.759261 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.877375 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:09.904515 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:09.904745 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:09.963972 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:09.969513 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:16:10.075776 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.095794 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:10.147213 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:10.204660 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:10.280282 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.320269 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.677572 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.684405 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:10.688727 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.701740 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.753268 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:10.834509 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:16:10.841509 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.847237 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.886529 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:10.977941 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:10.989477 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:11.043231 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:11.056710 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:11.074923 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:11.114701 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:11.207144 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:11.250229 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:11.277779 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:11.323140 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:11.525779 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:11.544736 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:11.590107 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 311: this message will repeat every 5th retry.
W20240503 02:16:11.619251 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:11.727026 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:11.832916 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:11.891391 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:11.965993 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:11.972828 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:12.039402 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:12.121073 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:12.160629 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:12.270320 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:12.335168 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:12.385344 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:12.412403 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:12.414491 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:12.542006 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:12.558796 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:12.559849 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:12.567942 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:12.616958 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:12.684952 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:12.885150 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.003700 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:13.123082 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:13.164320 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.219907 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.236181 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.302392 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.360270 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.378574 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:13.437722 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.445644 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:13.501667 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.603116 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:13.603195 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:13.664827 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.694422 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.848177 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:16:13.872135 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.911537 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:13.947501 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:13.986419 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:14.021441 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 316: this message will repeat every 5th retry.
W20240503 02:16:14.097261 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:14.147578 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:14.151343 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.184800 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:16:14.194916 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.206247 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.313433 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.381487 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:16:14.382511 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.485487 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:14.575063 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:14.576184 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.732765 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.809849 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.818262 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.840251 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.843377 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:14.870798 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:14.876704 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:14.900938 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:14.963261 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:15.056520 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:15.185748 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:16:15.232802 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:15.446183 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:15.455732 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:15.573964 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:15.584347 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:15.660744 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:15.665297 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:15.723912 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:15.831939 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:15.882274 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:15.908699 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:16:15.909843 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:15.921877 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:15.938390 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:15.962818 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:16.063139 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:16.114015 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:16.214157 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:16.244900 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:16.467433 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:16.475487 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:16.499028 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:16.522343 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 321: this message will repeat every 5th retry.
W20240503 02:16:16.598474 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:16.640859 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:16.687772 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:16.708671 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:16.740162 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:16.854351 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:16.988940 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:17.046468 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:17.097179 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:17.165766 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:17.199287 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:17.268416 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:17.312804 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:17.349264 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:17.375813 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:17.518738 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:17.519667 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:17.539875 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:17.590548 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:17.749784 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:17.896538 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:17.915042 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:17.920027 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:18.062144 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.117029 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.196846 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.240648 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.298120 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:18.356496 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.369762 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:18.404987 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.415776 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.421768 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.489529 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:18.526218 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:18.560355 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:18.748672 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.807490 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:18.810199 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.876305 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:18.975505 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:16:18.976560 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:18.991371 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.049149 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 326: this message will repeat every 5th retry.
W20240503 02:16:19.251272 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.267554 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:19.268836 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.398764 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:19.414219 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.422411 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:19.435652 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:16:19.531149 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.560459 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.563861 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:19.568022 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.643482 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.720767 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:16:19.833518 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.855980 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:19.871165 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:19.875492 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:19.969957 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.017304 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:20.160964 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:20.209071 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:20.228814 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:16:20.346855 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.381294 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:20.505409 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:20.559440 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.601882 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.758915 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.787519 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.797505 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.804011 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.830714 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:20.937425 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:20.945576 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:21.074788 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.080760 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:16:21.081776 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:21.091487 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.211663 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:21.214673 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.346896 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:21.387440 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:21.412045 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.502024 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 331: this message will repeat every 5th retry.
W20240503 02:16:21.606248 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:21.716624 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.750152 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.818758 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.838672 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.957772 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:21.977844 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:22.041285 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:22.056900 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:22.073537 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:22.078912 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:22.230264 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:22.298199 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:22.385460 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:22.406198 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:22.407953 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:22.561988 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:22.572870 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:22.614275 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:22.687925 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:22.703418 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:22.887984 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:22.957334 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:22.968384 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.043761 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.053457 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.095501 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:23.297209 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.308014 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:23.350831 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.359517 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.371791 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.528782 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.656388 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:23.695855 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:23.715781 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:23.735342 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.807138 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:23.893074 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 336: this message will repeat every 5th retry.
W20240503 02:16:23.954552 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:23.986979 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:16:24.027057 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.121110 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.155335 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:24.240939 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:24.323460 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:16:24.326181 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.360747 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:24.373498 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.413995 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.476994 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:16:24.557293 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.589612 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.626694 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.696991 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.704640 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:24.721571 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:24.769528 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:16:24.781703 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:24.810119 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:24.830572 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:25.007901 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:25.031975 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:25.091634 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:25.153431 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:25.167927 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:25.223146 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:25.244968 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:25.269140 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:16:25.522806 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:25.524173 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:25.557538 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:25.774855 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:25.781989 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:25.784368 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:25.841893 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:25.842692 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:26.010654 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:26.072218 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:26.120270 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:16:26.123224 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:26.151304 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:26.348160 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:26.435480 13246 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 49526us
W20240503 02:16:26.447286 10029 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.70:43707, user_credentials={real_user=slave}} blocked reactor thread for 44524.3us
W20240503 02:16:26.455618 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:26.479995 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:26.519888 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:26.592290 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 341: this message will repeat every 5th retry.
W20240503 02:16:26.594111 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:26.652878 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:26.672837 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:26.682406 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:26.694640 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:26.920518 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:26.922070 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:26.986300 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:26.992947 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.069231 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.119244 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:27.188656 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.264849 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.326936 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:27.360610 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.382144 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.424646 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:27.433390 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.454033 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:27.659205 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.697427 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:27.798583 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:27.801411 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:27.815629 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:27.893500 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.096073 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:28.192823 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.207975 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.287367 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.354671 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:28.399962 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.418751 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:16:28.510963 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.548413 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:28.575749 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.690897 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.730424 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.842270 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:28.876580 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:28.884548 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:28.986541 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.015229 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:16:29.016377 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.056946 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.110060 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.312520 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 346: this message will repeat every 5th retry.
W20240503 02:16:29.327278 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.347645 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:16:29.348968 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.414172 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:29.424860 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:29.435318 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:29.517895 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:16:29.520314 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.522059 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:29.551103 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:29.584925 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.643237 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.678489 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.876986 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.934060 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:29.944054 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:29.945252 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:29.953482 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:29.954406 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:30.003679 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:30.284580 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 02:16:30.285760 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:30.342048 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:30.368757 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:30.475236 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:30.479316 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:30.588888 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:30.615908 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:30.832423 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:30.854945 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:30.861945 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:30.863837 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:30.918028 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:31.026578 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:31.091320 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:31.250499 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:31.258973 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:31.267268 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:31.281683 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:31.335599 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:31.459525 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:31.510138 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:31.570415 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:31.696360 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:31.707895 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:31.715484 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:31.738060 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:31.802322 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:31.815461 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:31.830734 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:32.045289 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 351: this message will repeat every 5th retry.
W20240503 02:16:32.046379 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:32.060555 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:32.149370 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:32.185662 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:32.198959 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:32.230322 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:32.240623 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:32.264041 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:32.302510 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:32.314038 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:32.342941 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:32.608165 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:32.838223 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:32.894804 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:32.947736 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.128060 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.132658 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:33.147351 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.202823 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.246495 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:33.302737 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:33.435513 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:33.504554 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:33.517524 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.537614 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.627182 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.633634 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:33.671012 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:33.743075 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.747716 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:33.917912 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:33.998467 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.061161 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 86 similar messages]
W20240503 02:16:34.114214 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:34.128325 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.134673 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.135284 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.152904 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.208722 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.224416 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:34.266913 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:34.362810 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 356: this message will repeat every 5th retry.
W20240503 02:16:34.415048 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:16:34.417208 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.418848 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:34.436334 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.454996 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:34.461177 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:34.536693 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:16:34.537643 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.670488 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.733000 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:34.782863 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:34.788578 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.835269 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:34.889647 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:34.952431 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:16:35.225926 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:35.234496 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:35.336349 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:16:35.376629 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:35.519696 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:35.529953 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:35.584373 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:35.691112 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:35.799147 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:35.820315 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:35.829272 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:35.844667 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:35.899325 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:35.937927 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:36.108839 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:36.169234 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:36.282581 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:36.320816 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 41 similar messages]
W20240503 02:16:36.322011 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:36.379612 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:36.384187 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:36.431826 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:36.432348 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:36.661948 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:36.690032 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:36.720278 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:36.749294 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:36.754220 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:36.791224 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 361: this message will repeat every 5th retry.
W20240503 02:16:36.830675 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:36.858201 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:36.869683 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:36.968037 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.018215 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.026007 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:37.056731 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.088826 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.195472 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.240563 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:37.317886 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:37.327760 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.342772 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:37.364790 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.694557 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.816368 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.850163 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:37.926684 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.952244 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:37.980960 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:38.097963 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:38.291260 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:38.323508 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:38.416685 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:38.457252 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:38.464553 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:38.525799 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:38.565800 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:38.572700 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:38.577008 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:16:38.578408 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:38.617679 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:38.789433 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:38.903509 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:38.905820 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:38.922559 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:38.925001 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:38.940598 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:38.980760 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:39.020659 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:39.154330 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:16:39.272310 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:39.284957 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:39.346560 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:39.359426 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:39.368535 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:39.408337 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:39.437603 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 366: this message will repeat every 5th retry.
W20240503 02:16:39.461287 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:39.462474 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:39.496407 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:39.511979 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:39.550885 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 73 similar messages]
W20240503 02:16:39.578334 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:39.690644 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:39.691502 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:39.722481 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:39.838680 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:39.895201 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:39.928290 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:39.992130 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:40.085714 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:40.172423 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:40.202076 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:40.216272 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:40.254578 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:40.381626 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:16:40.382634 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:40.420172 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:40.533352 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:40.615986 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:40.775118 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:40.788172 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:40.822826 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:41.032783 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:41.039270 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:41.041123 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:41.042558 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:41.077757 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:41.109076 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:41.170114 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:41.211750 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:41.358630 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:16:41.382256 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:41.463152 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:41.465147 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:41.607160 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:41.683825 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:41.700690 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:41.706172 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:41.799233 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:41.839156 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:41.900328 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:41.913813 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.003242 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.013975 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:42.071367 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.141368 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:42.158987 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 371: this message will repeat every 5th retry.
W20240503 02:16:42.234933 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.276458 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.348412 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:42.379711 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.424901 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:42.594949 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.732841 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:42.853615 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.859930 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.881085 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:42.881260 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.947203 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:42.967607 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:43.012316 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:43.132194 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:43.333354 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:43.384706 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:43.385450 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:43.399490 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:43.454113 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:43.541776 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:43.542702 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:43.659792 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:43.700982 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:43.713948 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:43.754276 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:43.799703 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:43.876928 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:43.966372 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:44.040881 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.060555 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:44.189628 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:16:44.196121 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:44.288416 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:44.415930 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:44.435148 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.509239 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:44.538951 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.555598 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 02:16:44.560292 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:16:44.561944 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.631362 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.646440 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.676512 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.721828 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:44.724072 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 376: this message will repeat every 5th retry.
W20240503 02:16:44.770218 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:44.936012 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:44.942277 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.964972 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:44.974316 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:44.982906 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.205484 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:45.237644 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.266196 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.365738 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.496419 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:45.523773 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:16:45.524854 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.556268 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.556341 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.705853 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.832583 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:45.926070 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:45.973049 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:46.021530 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:46.033550 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:46.065317 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:46.076102 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:46.117452 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:46.139492 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:46.163686 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:46.389145 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:46.442067 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:46.442904 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:46.490963 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:46.531378 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:46.589828 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:46.695094 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:46.797914 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:46.822089 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.045542 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.051268 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:47.101433 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:47.210012 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.212284 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:47.294531 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.298476 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.304077 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.349774 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.426074 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:47.480412 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.499215 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 381: this message will repeat every 5th retry.
W20240503 02:16:47.515012 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:47.620096 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:47.626374 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.728153 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.775167 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.894279 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:47.924353 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:48.022723 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:48.156634 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:48.218070 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:48.303817 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:48.375988 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:48.376688 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:48.499727 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:48.530028 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:48.545013 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:48.545398 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:48.655488 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:48.690022 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:48.761050 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:48.771380 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:48.792624 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:48.853722 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:48.927111 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.029649 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:49.139245 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:49.161149 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:49.219441 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:16:49.341840 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.358194 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:49.426170 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.581076 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:16:49.582010 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.665827 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:49.754688 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:16:49.764019 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:49.765141 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.799988 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.807863 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.814064 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 386: this message will repeat every 5th retry.
W20240503 02:16:49.825057 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:49.839555 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.868588 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.879019 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:49.895023 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:49.940330 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:50.263327 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:50.270890 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 02:16:50.286578 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:50.322851 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:50.323220 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:50.424948 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:50.462325 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:50.515278 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:50.638612 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:50.639814 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:50.764328 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:50.840616 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:50.889983 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:50.895031 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:50.973367 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:51.009107 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:51.136016 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:51.170670 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:51.171135 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:51.184777 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:51.233474 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:51.257881 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:51.317086 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:51.330799 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:51.578770 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:51.620759 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:51.765935 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:51.872833 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:51.890799 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:51.916240 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:51.922981 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:51.936282 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:52.132329 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:52.154908 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 391: this message will repeat every 5th retry.
W20240503 02:16:52.291759 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.320410 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:52.333109 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:52.363795 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.369328 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.378435 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.393393 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.440351 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:52.441517 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:52.544602 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.561933 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.819844 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.842321 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.854173 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.868486 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:52.950706 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:53.023830 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:53.257126 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:53.327514 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:53.352888 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:53.380905 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:53.398566 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:53.485803 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:53.502238 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:53.530120 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:53.592973 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:53.655910 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:53.661729 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:53.773125 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:53.811539 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:53.866956 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:53.957522 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:53.959666 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:16:54.164044 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:54.303125 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 02:16:54.316157 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:54.365062 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:54.380810 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:54.434509 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:54.502795 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:54.580191 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:54.626402 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:16:54.627631 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:54.723850 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:54.732952 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 396: this message will repeat every 5th retry.
W20240503 02:16:54.742271 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:54.766109 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:54.766579 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:16:54.817597 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:54.841325 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:54.886381 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:54.898154 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:54.915334 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:54.986956 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:55.001523 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:55.015133 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:55.254709 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:55.312218 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:16:55.313181 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:55.421286 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:55.469817 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:55.492964 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:55.581501 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:55.599627 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:55.707257 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:55.734787 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:16:55.767881 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:55.930882 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:56.004506 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:56.007396 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:56.010510 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:56.122498 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:56.198449 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:56.261245 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:56.263787 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:56.287335 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:56.375696 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:56.435977 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:56.554862 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:56.574986 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:56.763851 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:16:56.814371 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:56.941975 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:56.945140 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:57.001439 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.152642 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:57.173208 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.181957 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:57.193527 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 401: this message will repeat every 5th retry.
W20240503 02:16:57.290640 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:57.332762 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.338169 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.377434 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.403633 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:57.446054 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.448191 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.460896 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.523298 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.573805 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.629041 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.640312 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:57.734227 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:57.872974 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:57.882586 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:58.034824 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:58.123088 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:58.176165 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:58.208226 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:58.289511 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:58.340679 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:58.399478 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:58.431337 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:58.505605 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:58.672282 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:58.693702 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:58.696807 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:58.774449 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:58.790866 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:58.840778 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:58.910884 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:59.100987 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:59.109253 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:59.178471 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:59.293634 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:59.298902 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:16:59.329510 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:16:59.441641 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:59.472782 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:59.488852 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:59.551494 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:59.700515 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:16:59.712479 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:16:59.774886 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:59.794632 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:16:59.817960 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:59.850307 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:59.866421 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:16:59.867228 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:16:59.887002 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:16:59.905757 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 406: this message will repeat every 5th retry.
W20240503 02:16:59.928225 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.011186 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.097389 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:00.122596 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.126701 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.152101 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.161830 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.205917 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:00.269033 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:00.518757 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:00.563112 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:00.653167 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:00.664323 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:00.700379 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.701121 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.874305 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:17:00.892294 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:00.924634 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:01.002306 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:01.005846 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:01.045823 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:01.071066 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:01.151794 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:01.154816 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:01.184309 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:01.308023 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:01.319269 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:01.491629 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:01.639361 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:01.748507 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:01.765000 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:01.912722 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:01.956141 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:01.965797 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:01.999120 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:17:02.125067 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:02.164889 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.202980 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.255854 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:02.331681 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:02.333194 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 411: this message will repeat every 5th retry.
W20240503 02:17:02.334439 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.407217 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.445587 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.483839 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.605906 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:02.641589 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.662943 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:02.663959 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:02.685961 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.719453 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.769337 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:02.823942 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:03.149233 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:03.176151 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:03.239603 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:03.347016 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:03.356496 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:03.381587 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:03.470784 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:03.477312 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:03.547152 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:03.621958 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:03.645112 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:03.651206 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:03.764678 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:03.868615 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:03.997098 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:04.001427 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:04.115866 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.255918 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:04.361488 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:17:04.408377 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 02:17:04.447264 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.482319 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:04.501363 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.514325 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:04.529207 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:04.572759 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:04.737154 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:17:04.738683 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.758988 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 416: this message will repeat every 5th retry.
W20240503 02:17:04.795307 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.858516 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.860499 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.893072 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:17:04.893910 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.900717 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:04.904379 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:05.029620 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:05.132066 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:05.132665 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:05.158632 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:05.298544 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:05.306715 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:05.378660 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:05.398182 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:05.432432 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:05.525864 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:05.617378 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:05.710572 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:05.731648 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:05.796406 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:05.798131 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:05.828073 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:05.882486 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:17:05.925128 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:06.003499 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:06.003808 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:06.090900 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:06.099905 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:06.100594 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:06.258844 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:06.345971 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:06.404786 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:06.565173 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:06.669847 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:06.738802 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:06.779120 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:06.825918 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:06.938215 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.039626 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.070060 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:07.117381 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:07.118176 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:07.198346 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.207765 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.247682 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.256589 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.295907 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 421: this message will repeat every 5th retry.
W20240503 02:17:07.371361 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:07.391338 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.415992 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.464015 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.547341 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:07.604198 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.661300 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:07.691908 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:07.763134 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:07.817471 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:07.827569 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:07.833021 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:08.112354 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:08.180218 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:08.252053 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:08.328608 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:08.361471 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:08.469497 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:08.473978 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:08.479100 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:08.491560 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:08.522950 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:08.552011 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:08.585716 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:08.844224 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:08.874732 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:08.982334 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:09.002077 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:09.013696 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:09.159878 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:09.211604 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:09.213232 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:09.247303 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:09.263504 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:09.411650 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:09.462333 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:09.491541 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:17:09.523275 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:09.576135 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:09.585353 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:17:09.586284 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 426: this message will repeat every 5th retry.
W20240503 02:17:09.588194 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:09.882795 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:09.883720 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:09.908972 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:09.930202 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:09.931386 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:09.964027 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:17:10.063099 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:10.087332 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.101356 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:10.129534 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.205302 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:10.246037 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.246105 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:10.322918 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:10.330168 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.343596 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.345181 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.738361 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.742190 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:17:10.764866 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.877600 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:10.919462 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:10.993834 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.116533 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.130923 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.178182 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:17:11.179251 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.225342 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:11.266479 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:11.268494 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.346954 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:11.365569 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:11.365903 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:11.546067 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.599046 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:11.769151 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.829528 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.844239 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:11.873792 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.877689 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:11.886760 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:11.897919 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:12.144961 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:12.169734 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:12.218917 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:12.236227 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:12.255236 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 431: this message will repeat every 5th retry.
W20240503 02:17:12.260932 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:12.356305 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:12.426164 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:12.459170 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:12.520061 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:12.566303 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:12.647594 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:12.748821 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:12.756264 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:12.834270 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:12.915333 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:12.933583 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:12.945426 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:13.048332 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:13.097137 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:13.336072 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:13.439286 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:13.441474 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:13.574934 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:13.596194 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:13.622828 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:13.702653 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:13.733439 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:13.852919 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:13.876878 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:13.901276 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:13.983955 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:14.023617 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:14.132750 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:14.212285 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:14.261219 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:14.362563 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:14.427685 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:14.444150 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:14.473074 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:14.532289 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:14.551046 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:14.714541 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 95 similar messages]
W20240503 02:17:14.731083 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:14.744679 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:14.744877 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:14.745790 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 436: this message will repeat every 5th retry.
W20240503 02:17:14.773680 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:14.828063 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:14.893056 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:14.964130 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:15.011317 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:17:15.013603 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:15.016314 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.053545 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:17:15.132887 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:15.169627 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.188391 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:15.222936 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.349629 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:15.425189 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:15.504606 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.518313 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.675299 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.706156 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.764612 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:15.832973 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.840617 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:17:15.912736 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:15.933619 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:15.945581 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:15.957484 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:16.202808 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:16.253382 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:16.304919 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:16.320333 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:16.422550 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:16.464174 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:16.479629 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:16.650307 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:16.663524 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:16.697440 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:16.717947 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:16.758152 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:16.834151 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.068658 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:17.086622 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 441: this message will repeat every 5th retry.
W20240503 02:17:17.092895 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:17.125982 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.130434 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.147217 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.250015 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.298703 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:17.361092 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.376238 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:17.412634 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.418354 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:17.477604 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:17.513963 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.517946 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:17.551764 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:17.567986 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:17.586093 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:17.849717 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:17.957973 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:17.981348 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:18.049326 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:18.067302 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:18.171635 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:18.262252 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:18.269661 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:18.286031 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:18.481875 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:18.528590 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:18.589263 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:18.673992 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:18.784576 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:18.863862 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:18.934820 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:18.941668 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:18.990453 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:19.125285 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:19.319013 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:19.327520 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:19.343596 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:19.370894 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:19.377903 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:19.448220 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:19.501124 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:19.520977 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 446: this message will repeat every 5th retry.
W20240503 02:17:19.533398 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:19.715008 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:17:19.723961 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:19.738857 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:19.753916 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:19.756909 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:19.807180 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:19.809787 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:19.829615 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:19.831430 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:19.851723 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:19.963415 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:20.070536 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:17:20.163280 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:20.187593 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:20.188669 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:20.188732 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:20.239774 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:20.240763 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:20.289994 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:20.424131 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:20.460382 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:20.474612 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:20.673297 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:20.687193 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:20.699782 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:20.856657 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:20.886274 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:20.941448 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:21.029695 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:17:21.053617 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:21.096534 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:21.166446 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:21.286505 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:17:21.334425 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:21.436139 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:21.444255 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:21.524907 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:21.552907 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:21.645553 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:21.709177 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:21.750005 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:21.845692 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:21.861173 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:21.867015 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:21.867038 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:21.930550 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:21.938081 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:21.995029 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 451: this message will repeat every 5th retry.
W20240503 02:17:22.061791 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:22.065371 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:22.088128 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.161897 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.218448 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.321601 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:22.399711 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.494683 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 43 similar messages]
W20240503 02:17:22.495786 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:22.612010 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.673168 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.681547 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:22.685122 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.728806 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.929255 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:22.930771 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:23.041582 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:23.074517 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:23.135591 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:23.142880 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:23.161758 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:23.331171 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:23.424080 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:23.438522 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:23.563462 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:23.676927 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:23.847363 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:23.947983 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.012868 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.077694 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.084360 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:24.175172 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.205119 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.220523 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.221737 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:24.317950 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:24.321210 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:24.391934 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:24.396087 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.531322 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:24.542136 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.587260 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:24.611853 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 456: this message will repeat every 5th retry.
W20240503 02:17:24.620654 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:24.749866 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:24.786789 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.807570 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 92 similar messages]
W20240503 02:17:24.851946 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:24.899598 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.919982 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:24.959465 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:25.055224 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:25.093472 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:17:25.127840 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:25.129585 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:25.242547 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:25.242909 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:17:25.262060 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:25.364424 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:25.583842 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:25.619542 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:25.631822 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:25.643111 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:25.684163 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:25.710536 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:25.723793 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:25.750725 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:25.912153 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:26.002516 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:26.050602 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:17:26.097010 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:26.098416 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:26.300354 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:26.429037 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:26.476263 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:26.477664 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:26.593148 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:26.606474 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:26.626305 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:26.635856 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:26.645146 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:26.778846 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:26.869189 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:26.889235 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:26.928241 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:26.942867 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.020025 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:27.172493 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:27.205925 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:27.250677 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:27.252856 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 461: this message will repeat every 5th retry.
W20240503 02:17:27.358738 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.420295 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.425886 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.490231 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.512187 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:27.530483 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:27.615278 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.679216 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:17:27.705297 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:27.739578 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.824359 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.865267 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:27.941216 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:27.986027 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:28.064850 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:28.151156 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:28.261037 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:28.281910 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:28.287431 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:28.294746 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:28.314147 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:28.518611 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:28.715657 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:28.723742 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:28.829905 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:28.938632 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:28.953775 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:29.021975 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:29.207284 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:29.208988 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:29.300308 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:29.313711 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:29.332969 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:29.338946 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:29.449756 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:29.461354 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:29.467613 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:29.487316 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:29.502003 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:29.728678 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:29.771879 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:29.780700 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:29.785044 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:29.809285 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:29.847455 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 466: this message will repeat every 5th retry.
W20240503 02:17:29.874761 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 02:17:29.989951 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.016332 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.081364 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:30.126967 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:30.173278 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:17:30.281440 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.289925 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:30.291574 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:17:30.309451 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.340314 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:30.425364 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:30.454524 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.576431 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.598238 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.677381 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:30.736827 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:30.758816 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.805836 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:30.848245 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:30.854420 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:31.090035 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:31.209779 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:31.266763 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:31.284399 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:31.316567 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:31.468277 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:31.520421 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:17:31.551483 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:31.565652 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:31.651050 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:31.669364 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:31.706681 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:31.818749 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:31.855252 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:31.956378 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:31.979212 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:32.059000 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.095134 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:32.128820 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:32.137619 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.157713 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:32.194260 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.217404 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:32.272264 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.357771 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.362769 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.547129 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 471: this message will repeat every 5th retry.
W20240503 02:17:32.632048 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:32.641177 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.755614 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:32.775424 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:17:32.776486 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:32.795136 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:32.820989 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.868075 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:32.993734 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:33.085649 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:33.102190 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:33.105823 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:33.149515 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:33.183624 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:33.232019 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:33.243876 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:33.452611 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:33.520203 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:33.616079 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:33.662865 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:33.718825 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:33.994427 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.039017 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.053066 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.078817 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.108553 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:34.167338 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.237672 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:34.264462 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:34.417207 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.435025 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:34.485702 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:34.505149 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.510859 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:34.570693 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:34.583550 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:34.711277 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.728691 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.750707 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:34.756938 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:34.901479 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:17:34.996481 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:35.086175 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:35.175223 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:17:35.191581 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.194537 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.280344 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 476: this message will repeat every 5th retry.
W20240503 02:17:35.314185 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.346185 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:35.347267 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:35.352149 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:35.369167 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:35.380128 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.396567 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.426054 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.429590 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:35.464020 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.510526 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:35.712425 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.790369 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.918382 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:35.919826 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:35.951416 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:35.978240 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:36.096326 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:36.118976 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:36.382181 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:36.417963 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:17:36.501303 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:36.532537 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:36.541428 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:36.652964 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:36.654675 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:36.673478 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:36.691753 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:36.693630 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:36.770975 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:36.789772 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:36.791854 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:36.831804 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:36.863978 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:36.976188 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:37.044365 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:37.079154 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:37.113411 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:37.160066 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:37.339213 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:37.446283 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:37.569056 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:37.645498 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:37.680959 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:37.689533 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:37.701830 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:37.822302 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:37.848536 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 481: this message will repeat every 5th retry.
W20240503 02:17:37.854135 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:17:37.876116 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:37.934842 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:37.937175 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:37.992404 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:38.002099 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:38.142835 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:38.149365 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:38.164952 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:38.249667 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:38.319042 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:38.399057 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:38.414032 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:38.414602 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:38.600261 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:17:38.601717 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 5547b33324db46c886287b4ee5035e43: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:38.627043 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:38.693203 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:38.760270 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:38.816882 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:38.894857 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:39.103065 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:39.174438 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:39.211546 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:39.216046 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:39.221022 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:39.251818 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:39.311909 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:39.332952 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:39.404565 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:39.464076 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:39.528604 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:39.575189 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:39.584491 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:39.801661 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:39.922390 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 02:17:39.940042 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:39.979962 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet 35aea508f59f4348b05d93a5afdde4ba: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:39.983209 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:40.022919 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.048100 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:40.115842 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.199589 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.238449 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:40.287899 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:17:40.384405 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.389294 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:40.418633 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:40.450037 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:40.451184 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:40.451153 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 486: this message will repeat every 5th retry.
W20240503 02:17:40.489611 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.532264 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.597754 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.658483 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:40.731761 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.750710 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.774463 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:40.913915 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:40.938300 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:41.005759 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:41.031574 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:41.035224 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:41.114712 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:41.254719 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:41.321024 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:41.571368 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:41.580682 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:17:41.651724 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:41.667553 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:41.692931 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:41.714151 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:41.731537 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:41.854547 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:41.856196 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:41.866091 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:41.962855 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:41.979625 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.012701 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.052188 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:42.134640 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.173995 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:42.174679 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:42.315744 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.342660 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.413697 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:42.448213 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:42.504163 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.536525 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:42.632092 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:42.764873 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.901753 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.907192 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.944342 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:42.950320 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:42.954663 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:43.052148 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:43.066257 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:43.117583 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 491: this message will repeat every 5th retry.
W20240503 02:17:43.249558 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:43.301460 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:43.345041 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:43.407500 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:43.415271 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:43.468494 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:43.546751 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:43.630985 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:43.631156 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:43.639158 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:43.879307 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:43.976277 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:43.995656 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:44.006397 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:44.062134 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:44.175259 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:44.261883 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:44.313083 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:44.328406 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:44.391476 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:44.398118 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:44.465960 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:44.538143 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:44.572898 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:44.692862 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:44.697155 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:44.784179 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:44.868741 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:44.880852 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:44.956923 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:44.969213 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:44.984306 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:17:44.986250 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:45.187961 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:45.239923 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.293504 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.332077 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:17:45.333220 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.371726 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 496: this message will repeat every 5th retry.
W20240503 02:17:45.432420 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:17:45.461699 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:45.542188 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:17:45.559491 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:45.565264 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.599896 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.668143 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.756584 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:45.827064 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.924587 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.925254 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:45.927668 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.042982 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:46.063848 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.112545 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.161322 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.358466 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.461009 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:46.474675 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:46.518388 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.555632 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.558737 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:46.612650 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:17:46.613986 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.667600 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.710662 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:46.716302 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:46.787022 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:17:46.795274 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:46.796159 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:46.889394 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:46.911123 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:46.992060 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:47.244033 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:47.280189 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:47.344954 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:47.352612 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:47.386734 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:47.395694 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:47.522243 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:47.558696 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:47.594381 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:47.720579 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:47.832795 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:47.839179 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:48.051635 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 501: this message will repeat every 5th retry.
W20240503 02:17:48.052840 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 42 similar messages]
W20240503 02:17:48.073347 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:48.120525 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:48.161863 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:48.178839 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:48.190814 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:48.223709 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:48.411983 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:48.417932 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:48.496436 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:48.514478 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:48.564638 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:48.608260 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:48.773655 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:48.801532 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:48.819880 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.009054 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:49.045660 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:49.184166 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:49.222865 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.224015 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.224822 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.237345 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:49.264816 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.273118 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:49.283581 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.283761 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.362393 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:49.415539 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:49.602684 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:49.626803 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.720424 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:49.856235 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:49.957039 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:49.966058 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:50.004638 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.044340 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:17:50.087575 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.113801 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.181861 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:50.385771 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:17:50.390493 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:50.399291 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.404412 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.506327 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:17:50.589725 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.651355 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.666528 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.712383 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:50.826066 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.858700 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:50.872787 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:17:50.874387 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 506: this message will repeat every 5th retry.
W20240503 02:17:50.897732 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:50.995939 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:51.004995 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.040716 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.101259 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.205355 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.286044 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:51.340710 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.365875 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:51.528179 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:51.589087 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.709596 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.709699 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.735327 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:17:51.736307 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:51.736955 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.765009 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:51.776687 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:51.778589 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:51.809463 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:51.826131 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:51.872304 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:51.887084 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:17:52.035986 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:52.128088 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:52.172199 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:52.245126 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:52.418469 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:52.451830 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:52.512370 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:52.525848 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:52.702786 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:52.722368 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:52.736342 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:52.817953 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:52.871243 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:52.923051 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:53.100081 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:53.141005 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:53.197742 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:53.239156 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:53.293843 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:53.365213 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 511: this message will repeat every 5th retry.
W20240503 02:17:53.394680 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:53.442689 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:53.444295 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:53.463199 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:53.591303 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:53.608592 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:53.717667 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:53.744503 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:53.772266 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:53.899762 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:53.932783 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:53.999650 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:54.035761 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.054440 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.144779 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.252472 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:54.265192 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.277412 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:54.321792 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.360081 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.563043 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:54.585482 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.587653 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:54.633168 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:54.666944 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:54.734163 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:54.912530 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.941411 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.962096 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:54.993389 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:55.044852 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:17:55.063287 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:55.124847 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:55.138809 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:55.169207 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:55.202064 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:55.305832 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:55.447312 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:17:55.593940 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:17:55.671556 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:55.717002 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:55.730998 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:55.791062 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:55.852581 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:55.877349 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:55.916002 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:17:55.917397 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 516: this message will repeat every 5th retry.
W20240503 02:17:56.006969 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.039919 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:56.149992 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:56.252938 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:56.296797 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.298367 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.339604 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:56.430454 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:56.504305 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.518698 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:56.533286 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:56.540633 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.628225 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.678589 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.763464 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:56.820099 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:56.884346 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.926374 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:17:56.927340 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:56.936841 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.021939 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:57.182214 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:57.232268 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:57.239777 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.283761 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:57.317770 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.342742 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.348862 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.370958 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:57.447698 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.510152 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.714386 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.740718 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:57.746613 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:57.923236 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:57.991909 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:58.042565 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:58.112879 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:58.129158 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:17:58.283968 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:58.478040 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:58.545881 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:58.563705 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:58.657791 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:58.673321 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 521: this message will repeat every 5th retry.
W20240503 02:17:58.676963 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:58.716392 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:58.756546 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:17:58.796859 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:58.807427 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:58.851080 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:59.069129 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.077358 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:17:59.109236 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.112668 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:59.147239 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:59.152434 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.258472 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.372643 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.457276 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:59.555135 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:59.583074 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.609112 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.749104 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:59.884717 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:59.898905 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.932780 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:17:59.935607 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:17:59.966579 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.969091 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:17:59.971674 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:00.008783 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:00.026062 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:00.033917 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:00.104303 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:00.137120 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:18:00.254371 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:00.321357 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:00.364475 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:00.448297 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 66 similar messages]
W20240503 02:18:00.595584 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:00.735252 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:00.758471 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:00.808341 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:18:00.891415 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:00.906533 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:00.994767 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:01.047338 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:01.197697 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:01.229259 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:18:01.230690 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 526: this message will repeat every 5th retry.
W20240503 02:18:01.327857 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:01.343716 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:01.436305 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:01.485903 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:01.523535 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:01.532912 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:01.562994 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:01.631419 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:01.774418 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:01.821666 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:01.847096 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:18:01.890377 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:01.904934 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:01.917729 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:01.929215 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:01.982820 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:18:02.021508 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.054493 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.146122 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.309751 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:02.319224 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:02.320963 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.324142 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.414470 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:02.425581 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.506243 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:02.515280 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.556743 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.578461 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.663540 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.855072 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:02.899055 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:02.981031 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:03.080937 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:03.104293 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:03.107838 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:03.237674 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:03.293654 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:03.401285 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:03.493678 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:03.500838 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:03.505437 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:03.550026 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:03.573138 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 531: this message will repeat every 5th retry.
W20240503 02:18:03.617789 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:03.842060 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:03.877964 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:03.945969 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:03.968008 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:04.010417 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:04.015264 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.099228 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.225863 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.297005 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:04.445533 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.476222 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:04.495237 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:04.526360 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.531015 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.572962 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.584546 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.625607 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.754477 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:04.892359 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:04.899273 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:04.912063 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:04.965487 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:05.028417 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:05.079340 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:05.094647 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:05.148339 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 90 similar messages]
W20240503 02:18:05.227732 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:05.238864 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:05.405881 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:05.488513 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:05.518374 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 67 similar messages]
W20240503 02:18:05.519852 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:05.556838 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:05.686547 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:05.687271 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:05.981581 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:06.013360 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:18:06.058967 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:06.094437 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:06.160789 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:06.176745 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:06.206569 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 536: this message will repeat every 5th retry.
W20240503 02:18:06.256335 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:06.294965 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:06.313702 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:06.445632 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
W20240503 02:18:06.584146 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:06.648927 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:06.664096 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:06.685868 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:06.708539 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:06.711748 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:06.750890 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:06.752692 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:06.882877 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:06.906131 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:06.937295 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:07.023272 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:18:07.089501 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.090086 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.124565 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:07.141139 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:18:07.148941 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.185653 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.206780 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:07.340668 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:07.395905 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:07.458863 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:07.501061 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.576689 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.589596 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.716817 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.768476 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:07.777915 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:07.925752 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:07.958797 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:08.011869 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:08.058601 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:08.106473 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:08.158424 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:08.206425 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:08.317778 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 38 similar messages]
W20240503 02:18:08.375886 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:08.498435 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 541: this message will repeat every 5th retry.
W20240503 02:18:08.619200 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:08.716285 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:08.764430 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:08.788969 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:08.826467 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.037286 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:09.050195 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:09.076684 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:09.162770 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.211333 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.251055 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.259673 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:09.279168 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:09.317118 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.353268 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:09.392944 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.397567 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:09.414331 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:09.527944 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.560701 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:09.635450 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:09.666267 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.707587 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:09.771035 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.841085 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:09.852764 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:09.982405 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.020925 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:10.029218 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.081185 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.146114 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.201452 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 87 similar messages]
W20240503 02:18:10.203963 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.403064 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:10.420084 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.434935 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:10.545864 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 72 similar messages]
W20240503 02:18:10.598584 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.713291 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 546: this message will repeat every 5th retry.
W20240503 02:18:10.773321 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.780426 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:10.812685 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:10.863998 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:10.996107 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:11.082576 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:18:11.190554 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:11.283041 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:11.362969 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:11.402340 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:11.485731 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:11.491094 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:11.578907 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:11.595798 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 21 similar messages]
W20240503 02:18:11.606070 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:11.697721 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:11.763845 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:11.795274 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:11.839282 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:11.848330 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:11.890514 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:11.920172 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:11.924809 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:11.930691 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:11.963572 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:11.992128 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:12.036129 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:12.068897 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:12.104079 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:12.124348 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:12.203984 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:18:12.261206 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:18:12.368851 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:12.469637 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:12.529939 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:12.619073 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:12.654868 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:12.688879 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:12.699636 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:12.711352 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:12.802335 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:12.882681 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:13.004290 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:13.124213 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 551: this message will repeat every 5th retry.
W20240503 02:18:13.126366 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:13.198776 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:13.318616 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:13.352960 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:13.450898 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 40 similar messages]
W20240503 02:18:13.459404 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:13.537501 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:13.559872 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:13.619181 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:13.825304 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:13.886533 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:13.966478 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:13.991124 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:14.061465 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:14.061720 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:14.092140 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:14.125773 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:14.162649 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:14.251732 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:14.266536 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:14.278293 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:14.282912 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:14.284904 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:14.315753 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:14.449751 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:14.554610 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:14.627552 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:14.647442 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:14.726851 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:14.831615 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:14.857806 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.108675 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.127619 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.153337 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.162124 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:15.189039 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:15.224546 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.225404 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.324193 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 91 similar messages]
W20240503 02:18:15.340854 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.418021 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.602421 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 69 similar messages]
W20240503 02:18:15.690264 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:15.693105 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 556: this message will repeat every 5th retry.
W20240503 02:18:15.694571 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:15.700699 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:15.759500 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:15.786347 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.839093 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:15.863801 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:16.042516 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:16.103786 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:18:16.250466 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:16.293484 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:16.378010 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:16.389014 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:16.406642 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:16.525643 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:16.552682 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:16.556232 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:16.674522 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:16.707881 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:16.728760 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:18:16.758006 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:16.767014 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:16.849678 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:16.887336 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:16.898897 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.019405 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:17.158993 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.224612 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
W20240503 02:18:17.225589 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:17.313122 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:18:17.330296 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.344612 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.352329 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.383659 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:17.412815 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.449646 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:17.487501 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:17.630221 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.672293 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.695088 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.717085 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.750898 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.775594 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:17.898488 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:17.960173 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:18.195279 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 561: this message will repeat every 5th retry.
W20240503 02:18:18.244860 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:18.285156 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:18.337733 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:18.373819 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:18.410245 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:18.599067 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:18.625854 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:18.626750 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:18.676687 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:18.698500 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:18.768553 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:18.956081 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:18.989625 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:19.050071 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:19.120203 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:19.221246 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:19.260555 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:19.312736 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:19.326943 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:19.342494 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:19.396931 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:19.506390 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:19.571337 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:19.590126 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:19.592852 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:19.626471 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:19.634102 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:19.916169 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:19.946527 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:19.955546 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.021847 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.037165 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:20.047639 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:20.049438 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.203256 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.206287 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.250530 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.339984 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.415508 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 93 similar messages]
W20240503 02:18:20.416719 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:20.446022 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:20.463356 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.581913 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:20.663705 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 71 similar messages]
W20240503 02:18:20.726976 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.803375 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 566: this message will repeat every 5th retry.
W20240503 02:18:20.817611 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:20.993588 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:20.994637 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:21.088627 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:21.097213 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:21.117659 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:21.167887 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 52 similar messages]
W20240503 02:18:21.200765 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:21.201548 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:21.455505 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:21.574631 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:21.603269 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:21.653447 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:21.684723 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:21.689781 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:21.706493 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:21.769860 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 18 similar messages]
W20240503 02:18:21.781770 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:21.864398 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:21.926518 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:21.968883 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.002220 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:22.190451 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:22.212410 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:22.240868 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.241181 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 48 similar messages]
W20240503 02:18:22.294656 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.350095 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:22.354066 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:18:22.355089 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.401274 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.453095 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.562654 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.707978 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.744179 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.761070 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:22.787087 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.820304 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:22.946540 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:22.958150 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:23.090673 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:23.150578 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:23.168399 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 571: this message will repeat every 5th retry.
W20240503 02:18:23.209723 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:23.216164 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:23.254540 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:23.388844 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:23.616065 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:23.680516 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:23.776362 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:23.777529 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:23.798532 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:23.829699 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:23.897523 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:23.909318 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.016764 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:24.028599 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:24.088055 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.250268 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:24.258620 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.351913 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:24.366896 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.384321 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.472424 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:24.483273 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:24.517882 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:24.625527 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.650849 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.721072 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:24.824146 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:24.880388 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.884799 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:24.895633 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.920706 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:24.923178 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:25.108654 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:25.331702 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:25.418033 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:25.418298 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 88 similar messages]
W20240503 02:18:25.433835 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:25.447117 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:25.448908 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:25.555620 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:25.579095 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:25.623421 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:25.666786 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 68 similar messages]
W20240503 02:18:25.699025 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:25.811393 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:25.832363 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 576: this message will repeat every 5th retry.
W20240503 02:18:25.868939 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:26.000013 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:26.049670 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:26.165675 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:26.254648 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:26.300124 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:26.304520 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:26.379324 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 53 similar messages]
W20240503 02:18:26.383682 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:26.404743 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:26.547075 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:26.609078 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:26.712518 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:26.732568 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:26.753788 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:26.755884 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:26.771595 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 19 similar messages]
W20240503 02:18:26.778452 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:26.846238 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:26.997741 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:27.045764 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:27.048278 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.103749 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:27.143046 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.158991 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:27.345678 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.377787 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 47 similar messages]
W20240503 02:18:27.378942 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.399964 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
W20240503 02:18:27.460781 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:27.541940 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.563910 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.604168 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.621106 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.660355 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:27.701097 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.714306 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:27.791041 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:27.843360 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:27.888429 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:28.040810 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:28.187186 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:28.206945 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 581: this message will repeat every 5th retry.
W20240503 02:18:28.224363 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:28.239594 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:28.272642 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:28.461711 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:28.499683 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:28.555954 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:28.567016 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:28.737716 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:28.751245 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:28.792754 13245 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 39 similar messages]
W20240503 02:18:28.847218 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:28.852026 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:28.993386 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:29.048525 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:29.050920 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:29.152678 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:29.263162 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:29.378360 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:29.422485 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:29.432628 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
W20240503 02:18:29.471033 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:29.517681 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:29.612916 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:29.613304 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:29.636829 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:29.747068 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:18:29.921334 12975 consensus_peers.cc:485] T 90d885837d5e403890617f6ead1fd6c5 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:29.970886 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:30.003876 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.057842 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.067366 10426 consensus_peers.cc:485] T 390ad47a9a534abb81d59dc7604c72d3 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.083293 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.252146 12975 consensus_peers.cc:485] T d2d4b31cd7334ed6b18288c1b6d3f958 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:30.273198 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.291365 10161 consensus_peers.cc:485] T da288f76df0a409eb01ee431d3b83b49 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.347548 10028 consensus_peers.cc:485] T ee635c5c4d2c4befa0283cb8226f28d7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:30.396970 12842 consensus_peers.cc:485] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:18:30.445387 10028 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 89 similar messages]
W20240503 02:18:30.446867 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:18:30.449724 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
W20240503 02:18:30.479846 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.519286 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.684937 10161 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 70 similar messages]
W20240503 02:18:30.782668 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:18:30.782965 13109 consensus_peers.cc:485] T b33490610e3b4819a61ef4bbfe0c5942 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 586: this message will repeat every 5th retry.
I20240503 02:18:30.895424 26617 consensus_queue.cc:579] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.302s)
W20240503 02:18:30.898151 10426 consensus_peers.cc:485] T 4c206318e0524101a4daee638813e041 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.900090 13245 consensus_peers.cc:462] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Unable to start Tablet Copy on peer: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:18:30.922816 13305 consensus_queue.cc:237] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } }
I20240503 02:18:30.925688 13036 raft_consensus.cc:1261] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:18:30.925676 13166 raft_consensus.cc:1261] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 1 index: 3. (index mismatch)
I20240503 02:18:30.926298 26799 consensus_queue.cc:1035] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:18:30.926558 26784 consensus_queue.cc:1035] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20240503 02:18:30.931941 26799 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } }
I20240503 02:18:30.937793 9936 catalog_manager.cc:4997] ChangeConfig:REMOVE_PEER RPC for tablet af66d6e994ab4049964f4218e05ef886 with cas_config_opid_index 2: ChangeConfig:REMOVE_PEER succeeded (attempt 1)
I20240503 02:18:30.939054 13166 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } }
I20240503 02:18:30.943300 9951 catalog_manager.cc:5484] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 2 to 3, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:30.948189 13036 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.3: config changed from index 2 to 3, NON_VOTER 47f901e617024baaa1865d12261ef84f (127.4.5.67) evicted. New config: { opid_index: 3 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } }
W20240503 02:18:30.954859 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:30.958042 9938 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 2 similar messages]
W20240503 02:18:30.959558 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet af66d6e994ab4049964f4218e05ef886: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:18:30.961362 13305 consensus_queue.cc:237] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 3, Committed index: 3, Last appended: 1.3, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:18:30.966871 13040 raft_consensus.cc:1261] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:18:30.967478 13166 raft_consensus.cc:1261] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 3. Preceding OpId from leader: term: 1 index: 4. (index mismatch)
I20240503 02:18:30.967819 26784 consensus_queue.cc:1035] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
W20240503 02:18:30.969269 13245 consensus_peers.cc:485] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af66d6e994ab4049964f4218e05ef886. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:30.970703 26799 consensus_queue.cc:1035] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 4, Last known committed idx: 3, Time since last communication: 0.000s
I20240503 02:18:30.974166 26784 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:18:30.977355 13036 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:18:30.978920 9936 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet af66d6e994ab4049964f4218e05ef886 with cas_config_opid_index 3: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:30.981686 13166 raft_consensus.cc:2932] T af66d6e994ab4049964f4218e05ef886 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.4: config changed from index 3 to 4, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:18:30.988787 9951 catalog_manager.cc:5484] T af66d6e994ab4049964f4218e05ef886 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 3 to 4, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 4 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { replace: true } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
W20240503 02:18:30.991487 12842 consensus_peers.cc:485] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:31.010946 10161 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:18:31.032606 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
W20240503 02:18:31.039392 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:31.183625 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:31.241350 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:18:31.412062 26646 consensus_queue.cc:579] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.178s)
I20240503 02:18:31.440395 10089 consensus_queue.cc:237] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:18:31.452913 13166 raft_consensus.cc:1261] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:18:31.454123 26646 consensus_queue.cc:579] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.076s)
I20240503 02:18:31.454526 26680 consensus_queue.cc:1035] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20240503 02:18:31.456121 10026 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e8819ae761d24f9a9b6f5570a14cf3c3. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:31.464013 26791 consensus_queue.cc:579] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.258s)
W20240503 02:18:31.480847 10028 consensus_peers.cc:485] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:31.490801 12842 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 51 similar messages]
W20240503 02:18:31.493410 12842 consensus_peers.cc:485] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:18:31.540060 26873 consensus_queue.cc:579] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.225s)
I20240503 02:18:31.542891 26874 consensus_queue.cc:579] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.090s)
W20240503 02:18:31.547071 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:18:31.562088 26617 consensus_queue.cc:579] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.048s)
I20240503 02:18:31.569903 13305 consensus_queue.cc:237] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:18:31.586513 26871 consensus_queue.cc:579] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.062s)
I20240503 02:18:31.589159 26873 raft_consensus.cc:2932] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:18:31.601284 13174 raft_consensus.cc:2932] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:18:31.609090 26875 consensus_queue.cc:579] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.346s)
I20240503 02:18:31.621189 9951 catalog_manager.cc:5484] T e8819ae761d24f9a9b6f5570a14cf3c3 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index -1 to 2, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
W20240503 02:18:31.639876 13245 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:31.641880 12906 raft_consensus.cc:1261] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:31.643404 26784 consensus_queue.cc:1035] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:31.647203 26647 consensus_queue.cc:579] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.252s)
I20240503 02:18:31.652382 26784 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:18:31.657820 12906 raft_consensus.cc:2932] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:18:31.672041 26764 consensus_queue.cc:579] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.281s)
I20240503 02:18:31.674175 13036 consensus_queue.cc:237] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:18:31.682058 26873 consensus_queue.cc:579] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.072s)
I20240503 02:18:31.684273 9936 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 09468c67719244c18d200463892a4f84 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:31.688556 10487 raft_consensus.cc:1261] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:31.689563 26879 consensus_queue.cc:1035] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:31.690575 9952 catalog_manager.cc:5484] T 09468c67719244c18d200463892a4f84 P e3cbe34992dc4b77b9aa9e9a4c143e78 reported cstate change: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
W20240503 02:18:31.693926 13246 consensus_peers.cc:485] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09468c67719244c18d200463892a4f84. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:31.708580 9938 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e8819ae761d24f9a9b6f5570a14cf3c3 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:31.712697 13040 consensus_queue.cc:237] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:18:31.721956 26763 consensus_queue.cc:579] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.025s)
W20240503 02:18:31.724359 12842 consensus_peers.cc:485] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:18:31.727959 26852 consensus_queue.cc:579] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.181s)
I20240503 02:18:31.735563 10091 consensus_queue.cc:237] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:18:31.742512 13174 raft_consensus.cc:1261] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:18:31.743431 26881 consensus_queue.cc:1035] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:18:31.747560 26797 consensus_queue.cc:579] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.118s)
I20240503 02:18:31.750603 13174 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:31.747704 26680 raft_consensus.cc:2932] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:31.766505 10091 consensus_queue.cc:237] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:18:31.769402 26862 consensus_queue.cc:579] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.492s)
I20240503 02:18:31.774808 9938 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 2fadc7e9c7604b5d8f7b8751b6ca78a7 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:31.780952 10224 consensus_queue.cc:237] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
W20240503 02:18:31.782330 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:31.782663 10028 consensus_peers.cc:485] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:31.785449 26617 consensus_queue.cc:579] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.121s)
W20240503 02:18:31.786324 10162 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:31.805091 26763 consensus_queue.cc:579] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.516s)
W20240503 02:18:31.808089 10161 consensus_peers.cc:485] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:31.809366 12842 consensus_peers.cc:485] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:18:31.824771 26877 ts_tablet_manager.cc:902] T af66d6e994ab4049964f4218e05ef886 P 031a0afb01b6469299fa829fa3467ef7: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:18:31.828202 26867 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:31.829337 26647 consensus_queue.cc:579] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.149s)
I20240503 02:18:31.832147 13296 consensus_queue.cc:237] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
W20240503 02:18:31.833847 13245 consensus_peers.cc:485] T 206e6a3850ff4242b30fa10c3f751fd2 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:31.834839 13109 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 20 similar messages]
I20240503 02:18:31.835675 9939 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 09e0373a72654a1082464fac8d428483 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:31.840548 9952 catalog_manager.cc:5484] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 4 to 5, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:31.842204 9951 catalog_manager.cc:5484] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P f93b103af083471b8e8c0376e9148aad reported cstate change: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:31.847579 10224 consensus_queue.cc:237] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:18:31.847883 10491 raft_consensus.cc:1261] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:31.848476 10491 raft_consensus.cc:2932] T 09e0373a72654a1082464fac8d428483 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:31.850996 26772 consensus_queue.cc:579] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.458s)
I20240503 02:18:31.865276 10089 raft_consensus.cc:1261] T ac9acb6fb9c940ce83922596c849c4d8 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:31.868351 26647 consensus_queue.cc:579] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.319s)
I20240503 02:18:31.876032 10072 tablet_service.cc:1429] Tablet server has 18 leaders and 0 scanners
I20240503 02:18:31.882423 26867 consensus_queue.cc:1035] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:18:31.885180 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 09e0373a72654a1082464fac8d428483. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:31.885362 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 87bb6c6f9ea646a5a26106312a3742eb. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:31.886505 12975 consensus_peers.cc:485] T 09e0373a72654a1082464fac8d428483 P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:31.886631 12975 consensus_peers.cc:485] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:31.892853 26772 consensus_queue.cc:1035] T ac9acb6fb9c940ce83922596c849c4d8 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:18:31.898406 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:18:31.921746 13019 tablet_service.cc:1429] Tablet server has 12 leaders and 0 scanners
W20240503 02:18:31.923049 10026 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.72:33671, user_credentials={real_user=slave}} blocked reactor thread for 333133us
I20240503 02:18:31.932468 10490 raft_consensus.cc:1261] T 818345467bd9400aa5fdd875b59f3249 P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:31.932762 26912 ts_tablet_manager.cc:902] T e8819ae761d24f9a9b6f5570a14cf3c3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
W20240503 02:18:31.943735 10161 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:31.946046 26873 consensus_queue.cc:1035] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:31.949357 13302 raft_consensus.cc:1261] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20240503 02:18:31.950810 26772 consensus_queue.cc:1035] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20240503 02:18:31.951581 26852 consensus_queue.cc:579] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.260s)
I20240503 02:18:31.955062 26791 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:31.975405 10226 consensus_queue.cc:237] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:18:31.978196 12906 raft_consensus.cc:1261] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 8ee3fd5ad3624ea1b9f681d24e9aabf2: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:31.978992 26799 consensus_queue.cc:1035] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:31.991955 9943 catalog_manager.cc:5484] T 87bb6c6f9ea646a5a26106312a3742eb P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 4 to 5, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:31.994369 26871 consensus_queue.cc:579] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.402s)
W20240503 02:18:31.997388 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c2a328e790e844b5aa9a1cf0fa16ee8c. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.000203 13245 consensus_peers.cc:485] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.010025 10225 consensus_queue.cc:237] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:18:32.028883 26799 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
I20240503 02:18:32.034252 10092 raft_consensus.cc:1261] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:32.037156 9936 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet c2a328e790e844b5aa9a1cf0fa16ee8c with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:32.035590 10090 consensus_queue.cc:237] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:18:32.046476 10094 consensus_queue.cc:237] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
W20240503 02:18:32.047433 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 818345467bd9400aa5fdd875b59f3249. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.047335 10092 consensus_queue.cc:237] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:18:32.048636 9939 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 87bb6c6f9ea646a5a26106312a3742eb with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:32.049870 26872 raft_consensus.cc:2932] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:18:32.056057 9938 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 0c183608a78b4313b844ae4498d95955 with cas_config_opid_index -1: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:32.055855 13302 raft_consensus.cc:2932] T 0c183608a78b4313b844ae4498d95955 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.2: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:18:32.057127 13040 raft_consensus.cc:1261] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer eff8e8d11a4447218741bb994aa41082: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:18:32.058370 26872 consensus_queue.cc:1035] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:18:32.058424 13036 consensus_queue.cc:237] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:18:32.041388 12906 raft_consensus.cc:2932] T c2a328e790e844b5aa9a1cf0fa16ee8c P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
W20240503 02:18:32.065515 10028 consensus_peers.cc:485] T 12da2248f416404da29a4557b90daef7 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 591: this message will repeat every 5th retry.
W20240503 02:18:32.065757 10028 consensus_peers.cc:485] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.069612 9943 catalog_manager.cc:5484] T c2a328e790e844b5aa9a1cf0fa16ee8c P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 4 to 5, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:32.070524 26772 consensus_queue.cc:579] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.521s)
I20240503 02:18:32.074213 13174 consensus_queue.cc:237] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
W20240503 02:18:32.074508 10162 consensus_peers.cc:485] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0c183608a78b4313b844ae4498d95955. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.074708 10162 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c014b2069ae42788aae87d8304fa155. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.079420 10091 consensus_queue.cc:237] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } }
I20240503 02:18:32.091097 26934 ts_tablet_manager.cc:902] T 09468c67719244c18d200463892a4f84 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671)
I20240503 02:18:32.095327 26772 consensus_queue.cc:1035] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Connected to new peer: Peer: permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:18:32.096827 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119): Couldn't send request to peer 031a0afb01b6469299fa829fa3467ef7. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 731649c884c3461ab562590f012727c5. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.097131 10161 consensus_peers.cc:485] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.110411 10426 consensus_peers.cc:485] T 5c762709f66f44b98089e69cbace1570 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
W20240503 02:18:32.120240 12975 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.125106 10491 consensus_queue.cc:237] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } }
I20240503 02:18:32.125747 26940 ts_tablet_manager.cc:902] T 87bb6c6f9ea646a5a26106312a3742eb P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
W20240503 02:18:32.131393 10029 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e94a48cad1d648b18d709b46566d810b. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.136382 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.138422 12908 raft_consensus.cc:1261] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:18:32.138557 10028 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.140002 13109 consensus_peers.cc:485] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0bee269120a44b53b5bacc722f6742a9. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.140954 26862 consensus_queue.cc:1035] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:18:32.144048 13036 raft_consensus.cc:1261] T 0bee269120a44b53b5bacc722f6742a9 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer f93b103af083471b8e8c0376e9148aad: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:32.145052 26778 consensus_queue.cc:1035] T 0bee269120a44b53b5bacc722f6742a9 P f93b103af083471b8e8c0376e9148aad [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20240503 02:18:32.145447 13153 tablet_service.cc:1429] Tablet server has 9 leaders and 0 scanners
I20240503 02:18:32.158533 26772 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
W20240503 02:18:32.160653 10028 consensus_peers.cc:485] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.166255 10029 consensus_peers.cc:485] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 -> Peer f93b103af083471b8e8c0376e9148aad (127.4.5.71:46557): Couldn't send request to peer f93b103af083471b8e8c0376e9148aad. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9abc528ca83640c3bf66022cf2c9b914. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.166311 13036 raft_consensus.cc:2932] T 731649c884c3461ab562590f012727c5 P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } } }
W20240503 02:18:32.172040 10161 consensus_peers.cc:485] T 20366a3e8dcc4996bfdd1d60ef1be077 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 596: this message will repeat every 5th retry.
I20240503 02:18:32.171701 26647 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:18:32.174513 10488 raft_consensus.cc:1261] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Refusing update from remote peer 8d454a3607d042dc9559ed59c3ccd84b: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20240503 02:18:32.174393 10490 raft_consensus.cc:2932] T 87bb6c6f9ea646a5a26106312a3742eb P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:32.178130 26867 consensus_queue.cc:1035] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20240503 02:18:32.187593 13039 raft_consensus.cc:1261] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20240503 02:18:32.188699 26874 consensus_queue.cc:1035] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:18:32.192000 13302 raft_consensus.cc:1261] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20240503 02:18:32.193275 26646 consensus_queue.cc:1035] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:18:32.196668 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.197145 10028 consensus_peers.cc:485] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.204811 26912 tablet_copy_client.cc:308] T e8819ae761d24f9a9b6f5570a14cf3c3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.65:38119
I20240503 02:18:32.209265 26874 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
W20240503 02:18:32.219539 10161 consensus_peers.cc:485] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.226027 13302 raft_consensus.cc:2932] T 9abc528ca83640c3bf66022cf2c9b914 P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
W20240503 02:18:32.227097 9938 outbound_call.cc:322] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.4.5.67:41899, user_credentials={real_user=slave}} blocked reactor thread for 93720.9us
I20240503 02:18:32.228482 9938 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 9abc528ca83640c3bf66022cf2c9b914 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:32.231612 10094 raft_consensus.cc:2932] T 4c014b2069ae42788aae87d8304fa155 P 031a0afb01b6469299fa829fa3467ef7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:18:32.232404 9938 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 731649c884c3461ab562590f012727c5 with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:32.232821 9938 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 4c014b2069ae42788aae87d8304fa155 with cas_config_opid_index 4: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 02:18:32.233245 9938 catalog_manager.cc:4573] TS 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): DeleteTablet:TABLET_DATA_TOMBSTONED RPC failed for tablet af66d6e994ab4049964f4218e05ef886: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111)
I20240503 02:18:32.234092 26852 consensus_queue.cc:579] T 254af82e74a045f8a9c9d9de48cdee3e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Leader has been unable to successfully communicate with peer 47f901e617024baaa1865d12261ef84f for more than 300 seconds (300.497s)
I20240503 02:18:32.235739 9951 catalog_manager.cc:5484] T 9abc528ca83640c3bf66022cf2c9b914 P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:32.236713 26879 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:18:32.245113 12887 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
I20240503 02:18:32.254711 9952 catalog_manager.cc:5484] T 0c183608a78b4313b844ae4498d95955 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index -1 to 2, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:32.257862 10115 tablet_copy_service.cc:140] P 031a0afb01b6469299fa829fa3467ef7: Received BeginTabletCopySession request for tablet e8819ae761d24f9a9b6f5570a14cf3c3 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 ({username='slave'} at 127.4.5.72:33685)
I20240503 02:18:32.258273 10115 tablet_copy_service.cc:161] P 031a0afb01b6469299fa829fa3467ef7: Beginning new tablet copy session on tablet e8819ae761d24f9a9b6f5570a14cf3c3 from peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 at {username='slave'} at 127.4.5.72:33685: session id = 8ee3fd5ad3624ea1b9f681d24e9aabf2-e8819ae761d24f9a9b6f5570a14cf3c3
I20240503 02:18:32.261766 10115 tablet_copy_source_session.cc:205] T e8819ae761d24f9a9b6f5570a14cf3c3 P 031a0afb01b6469299fa829fa3467ef7: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:18:32.265331 12905 consensus_queue.cc:237] T 67be061441154fbba1e227d32474cd5e P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:18:32.270388 12907 consensus_queue.cc:237] T f1f1661a75a74746a4485b88b8e5b185 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } }
I20240503 02:18:32.265386 12903 consensus_queue.cc:237] T 9fbc1a14c000465e9bb642fcc1e84611 P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: false } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
I20240503 02:18:32.279615 9943 catalog_manager.cc:5484] T 4c014b2069ae42788aae87d8304fa155 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 4 to 5, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:32.280608 9943 catalog_manager.cc:5484] T 731649c884c3461ab562590f012727c5 P eff8e8d11a4447218741bb994aa41082 reported cstate change: config changed from index 6 to 7, NON_VOTER 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65) added. New cstate: current_term: 1 leader_uuid: "eff8e8d11a4447218741bb994aa41082" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:32.281154 26874 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
I20240503 02:18:32.284762 26934 tablet_copy_client.cc:308] T 09468c67719244c18d200463892a4f84 P f93b103af083471b8e8c0376e9148aad: tablet copy: Beginning tablet copy session from remote peer at address 127.4.5.72:33671
I20240503 02:18:32.286144 9939 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 317ce62718cb412bbf04569cab14309f with cas_config_opid_index 7: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 02:18:32.288218 12976 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 96264.6us
I20240503 02:18:32.293716 10490 raft_consensus.cc:2932] T 317ce62718cb412bbf04569cab14309f P c6ec2f1283994a3089741715f811c850 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New config: { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } } }
I20240503 02:18:32.299129 9943 catalog_manager.cc:5484] T e94a48cad1d648b18d709b46566d810b P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:32.309427 9938 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet e94a48cad1d648b18d709b46566d810b with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
I20240503 02:18:32.311211 13330 tablet_copy_service.cc:140] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Received BeginTabletCopySession request for tablet 09468c67719244c18d200463892a4f84 from peer f93b103af083471b8e8c0376e9148aad ({username='slave'} at 127.4.5.71:58421)
I20240503 02:18:32.310487 10227 consensus_queue.cc:237] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: VOTER last_known_addr { host: "127.4.5.66" port: 42397 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: NON_VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: true } }
I20240503 02:18:32.311496 13330 tablet_copy_service.cc:161] P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Beginning new tablet copy session on tablet 09468c67719244c18d200463892a4f84 from peer f93b103af083471b8e8c0376e9148aad at {username='slave'} at 127.4.5.71:58421: session id = f93b103af083471b8e8c0376e9148aad-09468c67719244c18d200463892a4f84
I20240503 02:18:32.314893 13330 tablet_copy_source_session.cc:205] T 09468c67719244c18d200463892a4f84 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: Tablet Copy: opened 0 blocks and 1 log segments
I20240503 02:18:32.316712 26958 ts_tablet_manager.cc:902] T 2fadc7e9c7604b5d8f7b8751b6ca78a7 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer 031a0afb01b6469299fa829fa3467ef7 (127.4.5.65:38119)
I20240503 02:18:32.318847 26934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09468c67719244c18d200463892a4f84. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:18:32.333081 10426 consensus_peers.cc:485] T db2da39193164d1ebf830282258d47d7 P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 601: this message will repeat every 5th retry.
I20240503 02:18:32.347537 9943 catalog_manager.cc:5484] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b reported cstate change: config changed from index 7 to 8, NON_VOTER 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72) added. New cstate: current_term: 1 leader_uuid: "8d454a3607d042dc9559ed59c3ccd84b" committed_config { opid_index: 8 OBSOLETE_local: false peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: NON_VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:32.362872 12902 raft_consensus.cc:2932] T e94a48cad1d648b18d709b46566d810b P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER f93b103af083471b8e8c0376e9148aad (127.4.5.71) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } attrs { promote: false } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: NON_VOTER last_known_addr { host: "127.4.5.71" port: 46557 } attrs { promote: true } } }
W20240503 02:18:32.371133 12843 outbound_call.cc:322] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.4.5.71:46557, user_credentials={real_user=slave}} blocked reactor thread for 84086.9us
I20240503 02:18:32.376013 26955 ts_tablet_manager.cc:902] T 09e0373a72654a1082464fac8d428483 P eff8e8d11a4447218741bb994aa41082: Initiating tablet copy from peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707)
I20240503 02:18:32.382720 12904 consensus_queue.cc:237] T 3303da732fa142b482c7d530c413c5dc P e3cbe34992dc4b77b9aa9e9a4c143e78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 2 OBSOLETE_local: false peers { permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 } } peers { permanent_uuid: "f93b103af083471b8e8c0376e9148aad" member_type: VOTER last_known_addr { host: "127.4.5.71" port: 46557 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: NON_VOTER last_known_addr { host: "127.4.5.65" port: 38119 } attrs { promote: true } }
W20240503 02:18:32.430426 10162 consensus_peers.cc:485] T 09a142900b5c4909ad16b4abc70ee952 P eff8e8d11a4447218741bb994aa41082 -> Peer 8d454a3607d042dc9559ed59c3ccd84b (127.4.5.70:43707): Couldn't send request to peer 8d454a3607d042dc9559ed59c3ccd84b. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.474429 26968 ts_tablet_manager.cc:902] T ac9acb6fb9c940ce83922596c849c4d8 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:18:32.482811 26951 ts_tablet_manager.cc:902] T 4c014b2069ae42788aae87d8304fa155 P f93b103af083471b8e8c0376e9148aad: Initiating tablet copy from peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397)
I20240503 02:18:32.487390 26878 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
F20240503 02:18:32.482662 26968 trace_metrics.cc:68] Check failed: g_intern_map->size() < 100 (100 vs. 100) Too many interned strings: (MaintenanceMgr.queue_time_us, MaintenanceMgr.queue_time_us) (MaintenanceMgr.run_cpu_time_us, MaintenanceMgr.run_cpu_time_us) (MaintenanceMgr.run_wall_time_us, MaintenanceMgr.run_wall_time_us) (apply.queue_time_us, apply.queue_time_us) (apply.run_cpu_time_us, apply.run_cpu_time_us) (apply.run_wall_time_us, apply.run_wall_time_us) (client-negotiator.queue_time_us, client-negotiator.queue_time_us) (client-negotiator.run_cpu_time_us, client-negotiator.run_cpu_time_us) (client-negotiator.run_wall_time_us, client-negotiator.run_wall_time_us) (compiler_manager_pool.queue_time_us, compiler_manager_pool.queue_time_us) (compiler_manager_pool.run_cpu_time_us, compiler_manager_pool.run_cpu_time_us) (compiler_manager_pool.run_wall_time_us, compiler_manager_pool.run_wall_time_us) (dirs.queue_time_us, dirs.queue_time_us) (dirs.run_cpu_time_us, dirs.run_cpu_time_us) (dirs.run_wall_time_us, dirs.run_wall_time_us) (dns-resolver.queue_time_us, dns-resolver.queue_time_us) (dns-resolver.run_cpu_time_us, dns-resolver.run_cpu_time_us) (dns-resolver.run_wall_time_us, dns-resolver.run_wall_time_us) (log-alloc.queue_time_us, log-alloc.queue_time_us) (log-alloc.run_cpu_time_us, log-alloc.run_cpu_time_us) (log-alloc.run_wall_time_us, log-alloc.run_wall_time_us) (prepare.queue_time_us, prepare.queue_time_us) (prepare.run_cpu_time_us, prepare.run_cpu_time_us) (prepare.run_wall_time_us, prepare.run_wall_time_us) (raft.queue_time_us, raft.queue_time_us) (raft.run_cpu_time_us, raft.run_cpu_time_us) (raft.run_wall_time_us, raft.run_wall_time_us) (repair_pool.queue_time_us, repair_pool.queue_time_us) (repair_pool.run_cpu_time_us, repair_pool.run_cpu_time_us) (repair_pool.run_wall_time_us, repair_pool.run_wall_time_us) (server-negotiator.queue_time_us, server-negotiator.queue_time_us) (server-negotiator.run_cpu_time_us, server-negotiator.run_cpu_time_us) (server-negotiator.run_wall_time_us, server-negotiator.run_wall_time_us) (tablet-copy.queue_time_us, tablet-copy.queue_time_us) (tablet-copy.run_cpu_time_us, tablet-copy.run_cpu_time_us) (tablet-copy.run_wall_time_us, tablet-copy.run_wall_time_us) (tablet-delete.queue_time_us, tablet-delete.queue_time_us) (tablet-delete.run_cpu_time_us, tablet-delete.run_cpu_time_us) (tablet-delete.run_wall_time_us, tablet-delete.run_wall_time_us) (tablet-download-pool-09468c67719244c18d200463892a4f84.queue_time_us, tablet-download-pool-09468c67719244c18d200463892a4f84.queue_time_us) (tablet-download-pool-09468c67719244c18d200463892a4f84.run_cpu_time_us, tablet-download-pool-09468c67719244c18d200463892a4f84.run_cpu_time_us) (tablet-download-pool-09468c67719244c18d200463892a4f84.run_wall_time_us, tablet-download-pool-09468c67719244c18d200463892a4f84.run_wall_time_us) (tablet-download-pool-0dc2733b737b431684ce439c3d3fa762.queue_time_us, tablet-download-pool-0dc2733b737b431684ce439c3d3fa762.queue_time_us) (tablet-download-pool-0dc2733b737b431684ce439c3d3fa762.run_cpu_time_us, tablet-download-pool-0dc2733b737b431684ce439c3d3fa762.run_cpu_time_us) (tablet-download-pool-0dc2733b737b431684ce439c3d3fa762.run_wall_time_us, tablet-download-pool-0dc2733b737b431684ce439c3d3fa762.run_wall_time_us) (tablet-download-pool-1578111fde6e43fd853142a48eb34ef5.queue_time_us, tablet-download-pool-1578111fde6e43fd853142a48eb34ef5.queue_time_us) (tablet-download-pool-1578111fde6e43fd853142a48eb34ef5.run_cpu_time_us, tablet-download-pool-1578111fde6e43fd853142a48eb34ef5.run_cpu_time_us) (tablet-download-pool-1578111fde6e43fd853142a48eb34ef5.run_wall_time_us, tablet-download-pool-1578111fde6e43fd853142a48eb34ef5.run_wall_time_us) (tablet-download-pool-274d302b083b42019a6f03305fe516e7.queue_time_us, tablet-download-pool-274d302b083b42019a6f03305fe516e7.queue_time_us) (tablet-download-pool-274d302b083b42019a6f03305fe516e7.run_cpu_time_us, tablet-download-pool-274d302b083b42019a6f03305fe516e7.run_cpu_time_us) (tablet-download-pool-274d302b083b42019a6f03305fe516e7.run_wall_time_us, tablet-download-pool-274d302b083b42019a6f03305fe516e7.run_wall_time_us) (tablet-download-pool-2fadc7e9c7604b5d8f7b8751b6ca78a7.queue_time_us, tablet-download-pool-2fadc7e9c7604b5d8f7b8751b6ca78a7.queue_time_us) (tablet-download-pool-2fadc7e9c7604b5d8f7b8751b6ca78a7.run_cpu_time_us, tablet-download-pool-2fadc7e9c7604b5d8f7b8751b6ca78a7.run_cpu_time_us) (tablet-download-pool-2fadc7e9c7604b5d8f7b8751b6ca78a7.run_wall_time_us, tablet-download-pool-2fadc7e9c7604b5d8f7b8751b6ca78a7.run_wall_time_us) (tablet-download-pool-36457442a8e24ece9a1c2d8cdab89b78.queue_time_us, tablet-download-pool-36457442a8e24ece9a1c2d8cdab89b78.queue_time_us) (tablet-download-pool-36457442a8e24ece9a1c2d8cdab89b78.run_cpu_time_us, tablet-download-pool-36457442a8e24ece9a1c2d8cdab89b78.run_cpu_time_us) (tablet-download-pool-36457442a8e24ece9a1c2d8cdab89b78.run_wall_time_us, tablet-download-pool-36457442a8e24ece9a1c2d8cdab89b78.run_wall_time_us) (tablet-download-pool-49ce473013a942698081c389ace54b32.queue_time_us, tablet-download-pool-49ce473013a942698081c389ace54b32.queue_time_us) (tablet-download-pool-49ce473013a942698081c389ace54b32.run_cpu_time_us, tablet-download-pool-49ce473013a942698081c389ace54b32.run_cpu_time_us) (tablet-download-pool-49ce473013a942698081c389ace54b32.run_wall_time_us, tablet-download-pool-49ce473013a942698081c389ace54b32.run_wall_time_us) (tablet-download-pool-5aa7ed62ebf44431a1a6db26ef73052a.queue_time_us, tablet-download-pool-5aa7ed62ebf44431a1a6db26ef73052a.queue_time_us) (tablet-download-pool-5aa7ed62ebf44431a1a6db26ef73052a.run_cpu_time_us, tablet-download-pool-5aa7ed62ebf44431a1a6db26ef73052a.run_cpu_time_us) (tablet-download-pool-5aa7ed62ebf44431a1a6db26ef73052a.run_wall_time_us, tablet-download-pool-5aa7ed62ebf44431a1a6db26ef73052a.run_wall_time_us) (tablet-download-pool-a2aac20a18f4444a9ff79642b23c89d9.queue_time_us, tablet-download-pool-a2aac20a18f4444a9ff79642b23c89d9.queue_time_us) (tablet-download-pool-a2aac20a18f4444a9ff79642b23c89d9.run_cpu_time_us, tablet-download-pool-a2aac20a18f4444a9ff79642b23c89d9.run_cpu_time_us) (tablet-download-pool-a2aac20a18f4444a9ff79642b23c89d9.run_wall_time_us, tablet-download-pool-a2aac20a18f4444a9ff79642b23c89d9.run_wall_time_us) (tablet-download-pool-ac9acb6fb9c940ce83922596c849c4d8.queue_time_us, tablet-download-pool-ac9acb6fb9c940ce83922596c849c4d8.queue_time_us) (tablet-download-pool-ceb177d6952e4c978c8be93b5b026952.queue_time_us, tablet-download-pool-ceb177d6952e4c978c8be93b5b026952.queue_time_us) (tablet-download-pool-ceb177d6952e4c978c8be93b5b026952.run_cpu_time_us, tablet-download-pool-ceb177d6952e4c978c8be93b5b026952.run_cpu_time_us) (tablet-download-pool-ceb177d6952e4c978c8be93b5b026952.run_wall_time_us, tablet-download-pool-ceb177d6952e4c978c8be93b5b026952.run_wall_time_us) (tablet-download-pool-da288f76df0a409eb01ee431d3b83b49.queue_time_us, tablet-download-pool-da288f76df0a409eb01ee431d3b83b49.queue_time_us) (tablet-download-pool-da288f76df0a409eb01ee431d3b83b49.run_cpu_time_us, tablet-download-pool-da288f76df0a409eb01ee431d3b83b49.run_cpu_time_us) (tablet-download-pool-da288f76df0a409eb01ee431d3b83b49.run_wall_time_us, tablet-download-pool-da288f76df0a409eb01ee431d3b83b49.run_wall_time_us) (tablet-download-pool-dd081871c430470b95fce8f19634dc2e.queue_time_us, tablet-download-pool-dd081871c430470b95fce8f19634dc2e.queue_time_us) (tablet-download-pool-dd081871c430470b95fce8f19634dc2e.run_cpu_time_us, tablet-download-pool-dd081871c430470b95fce8f19634dc2e.run_cpu_time_us) (tablet-download-pool-dd081871c430470b95fce8f19634dc2e.run_wall_time_us, tablet-download-pool-dd081871c430470b95fce8f19634dc2e.run_wall_time_us) (tablet-download-pool-ee635c5c4d2c4befa0283cb8226f28d7.queue_time_us, tablet-download-pool-ee635c5c4d2c4befa0283cb8226f28d7.queue_time_us) (tablet-download-pool-ee635c5c4d2c4befa0283cb8226f28d7.run_cpu_time_us, tablet-download-pool-ee635c5c4d2c4befa0283cb8226f28d7.run_cpu_time_us) (tablet-download-pool-ee635c5c4d2c4befa0283cb8226f28d7.run_wall_time_us, tablet-download-pool-ee635c5c4d2c4befa0283cb8226f28d7.run_wall_time_us) (tablet-download-pool-fc0a049eeb0047558d3c9eb370467c53.queue_time_us, tablet-download-pool-fc0a049eeb0047558d3c9eb370467c53.queue_time_us) (tablet-download-pool-fc0a049eeb0047558d3c9eb370467c53.run_cpu_time_us, tablet-download-pool-fc0a049eeb0047558d3c9eb370467c53.run_cpu_time_us) (tablet-download-pool-fc0a049eeb0047558d3c9eb370467c53.run_wall_time_us, tablet-download-pool-fc0a049eeb0047558d3c9eb370467c53.run_wall_time_us) (tablet-open.queue_time_us, tablet-open.queue_time_us) (tablet-open.run_cpu_time_us, tablet-open.run_cpu_time_us) (tablet-open.run_wall_time_us, tablet-open.run_wall_time_us) (txn-commit.queue_time_us, txn-commit.queue_time_us) (txn-commit.run_cpu_time_us, txn-commit.run_cpu_time_us) (txn-commit.run_wall_time_us, txn-commit.run_wall_time_us) (txn-participant-registration.queue_time_us, txn-participant-registration.queue_time_us) (txn-participant-registration.run_cpu_time_us, txn-participant-registration.run_cpu_time_us) (txn-participant-registration.run_wall_time_us, txn-participant-registration.run_wall_time_us) (txn-status-manager.queue_time_us, txn-status-manager.queue_time_us) (txn-status-manager.run_cpu_time_us, txn-status-manager.run_cpu_time_us) (txn-status-manager.run_wall_time_us, txn-status-manager.run_wall_time_us) (txn-status-tablet-reload.queue_time_us, txn-status-tablet-reload.queue_time_us) (txn-status-tablet-reload.run_cpu_time_us, txn-status-tablet-reload.run_cpu_time_us) (txn-status-tablet-reload.run_wall_time_us, txn-status-tablet-reload.run_wall_time_us) (wal-append.queue_time_us, wal-append.queue_time_us) (wal-append.run_cpu_time_us, wal-append.run_cpu_time_us) (wal-append.run_wall_time_us, wal-append.run_wall_time_us)
I20240503 02:18:32.490245 13036 raft_consensus.cc:2932] T 476f6e414dae4db3ba9a187adad75f8c P 8d454a3607d042dc9559ed59c3ccd84b [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
W20240503 02:18:32.494230 26877 thread.cc:636] tablet-download-pool-af66d6e994ab4049964f4218e05ef886 [worker] (thread pool) Time spent creating pthread: real 0.555s user 0.018s sys 0.042s
W20240503 02:18:32.494367 26877 thread.cc:602] tablet-download-pool-af66d6e994ab4049964f4218e05ef886 [worker] (thread pool) Time spent starting thread: real 0.555s user 0.018s sys 0.042s
I20240503 02:18:32.495304 9938 catalog_manager.cc:4997] ChangeConfig:ADD_PEER:NON_VOTER RPC for tablet 476f6e414dae4db3ba9a187adad75f8c with cas_config_opid_index 6: ChangeConfig:ADD_PEER:NON_VOTER succeeded (attempt 1)
W20240503 02:18:32.497164 10089 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.556s user 0.000s sys 0.010s
W20240503 02:18:32.497282 10089 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.556s user 0.000s sys 0.010s
I20240503 02:18:32.499565 13302 raft_consensus.cc:1261] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Refusing update from remote peer c6ec2f1283994a3089741715f811c850: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
W20240503 02:18:32.500535 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer eff8e8d11a4447218741bb994aa41082 (127.4.5.66:42397): Couldn't send request to peer eff8e8d11a4447218741bb994aa41082. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.500694 10426 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 50 similar messages]
I20240503 02:18:32.501842 9943 catalog_manager.cc:5484] T 476f6e414dae4db3ba9a187adad75f8c P 031a0afb01b6469299fa829fa3467ef7 reported cstate change: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "031a0afb01b6469299fa829fa3467ef7" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8d454a3607d042dc9559ed59c3ccd84b" member_type: VOTER last_known_addr { host: "127.4.5.70" port: 43707 } attrs { promote: false } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } health_report { overall_health: UNKNOWN } } }
I20240503 02:18:32.502626 26655 consensus_queue.cc:1035] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20240503 02:18:32.506769 12902 raft_consensus.cc:1261] T 46e5690a42d44500b3e6ca24d771affc P e3cbe34992dc4b77b9aa9e9a4c143e78 [term 1 FOLLOWER]: Refusing update from remote peer 031a0afb01b6469299fa829fa3467ef7: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20240503 02:18:32.507525 10028 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.508033 10026 consensus_peers.cc:485] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.508896 26878 consensus_queue.cc:1035] T 46e5690a42d44500b3e6ca24d771affc P 031a0afb01b6469299fa829fa3467ef7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3cbe34992dc4b77b9aa9e9a4c143e78" member_type: VOTER last_known_addr { host: "127.4.5.69" port: 33471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20240503 02:18:32.512010 10426 consensus_peers.cc:485] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 -> Peer 47f901e617024baaa1865d12261ef84f (127.4.5.67:41899): Couldn't send request to peer 47f901e617024baaa1865d12261ef84f. Status: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20240503 02:18:32.513907 10487 thread.cc:636] wal-append [worker] (thread pool) Time spent creating pthread: real 0.551s user 0.000s sys 0.010s
W20240503 02:18:32.514176 10487 thread.cc:602] wal-append [worker] (thread pool) Time spent starting thread: real 0.551s user 0.000s sys 0.010s
I20240503 02:18:32.523873 10470 tablet_service.cc:1429] Tablet server has 13 leaders and 0 scanners
W20240503 02:18:32.524720 12973 consensus_peers.cc:485] T 317ce62718cb412bbf04569cab14309f P 8d454a3607d042dc9559ed59c3ccd84b -> Peer 8ee3fd5ad3624ea1b9f681d24e9aabf2 (127.4.5.72:33671): Couldn't send request to peer 8ee3fd5ad3624ea1b9f681d24e9aabf2. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 317ce62718cb412bbf04569cab14309f. This is attempt 1: this message will repeat every 5th retry.
I20240503 02:18:32.527138 26764 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P c6ec2f1283994a3089741715f811c850 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:32.528256 26912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8819ae761d24f9a9b6f5570a14cf3c3. 1 dirs total, 0 dirs full, 0 dirs failed
W20240503 02:18:32.535204 12975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.4.5.67:41899: connect: Connection refused (error 111) [suppressed 49 similar messages]
I20240503 02:18:32.534844 13305 raft_consensus.cc:2932] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:32.549060 26912 tablet_copy_client.cc:783] T e8819ae761d24f9a9b6f5570a14cf3c3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 0 data blocks...
I20240503 02:18:32.549393 26912 tablet_copy_client.cc:647] T e8819ae761d24f9a9b6f5570a14cf3c3 P 8ee3fd5ad3624ea1b9f681d24e9aabf2: tablet copy: Starting download of 1 WAL segments...
I20240503 02:18:32.557610 9951 catalog_manager.cc:5484] T 73ae9bc7a96149259bd807369c2daacd P 8ee3fd5ad3624ea1b9f681d24e9aabf2 reported cstate change: config changed from index 6 to 7, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New cstate: current_term: 1 leader_uuid: "c6ec2f1283994a3089741715f811c850" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "8ee3fd5ad3624ea1b9f681d24e9aabf2" member_type: VOTER last_known_addr { host: "127.4.5.72" port: 33671 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082" member_type: NON_VOTER last_known_addr { host: "127.4.5.66" port: 42397 } attrs { promote: true } } }
I20240503 02:18:32.567114 26862 raft_consensus.cc:2932] T 818345467bd9400aa5fdd875b59f3249 P 031a0afb01b6469299fa829fa3467ef7 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index 4 to 5, NON_VOTER eff8e8d11a4447218741bb994aa41082 (127.4.5.66) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "031a0afb01b6469299fa829fa3467ef7" member_type: VOTER last_known_addr { host: "127.4.5.65" port: 38119 } } peers { permanent_uuid: "c6ec2f1283994a3089741715f811c850" member_type: VOTER last_known_addr { host: "127.4.5.68" port: 35025 } } peers { permanent_uuid: "47f901e617024baaa1865d12261ef84f" member_type: VOTER last_known_addr { host: "127.4.5.67" port: 41899 } attrs { promote: false } } peers { permanent_uuid: "eff8e8d11a4447218741bb994aa41082"